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The commi : : 
Ste) prs, a implement an Action Programme on Essential Drugs (APED) implies a 
inploaneicn, eters progress toward that goal. Progress can either be evaluated 

° e programme, i.e., on-going evaluation/ i 
: : : review d 
pletion of the programme, i.e., ex-post evaluation. ond ate 


On-going evaluation (sometimes called review) is an integral part of programm 
agement and is primarly done in order to analyse and propose Pec cona en ne £ 
‘lementation and operation, some of which may already have been identified p the * 
itoring system (see separate guidelines). The main purpose of ex-post Naieeetae is to 
‘lew the experience of the programme as a basis for future policy formulation and 
gramme design. The Guidelines in this document have been designed to accom lish on-goi 
well as ex-post evaluations. 5 , 


ee wat , 4 , : ae 
7 uation consists of measuring programme effectiveness and efficiency. Programme 
ectiveness 1s defined as an analysis of the results obtained in relation to the 

lectives of the programme, whereas programme efficiency is more an analysis of how 
ources are being spent in relation to cost and service provided. To state it in another 


/, programme effectiveness is concerned with the outcome, whereas programme efficiency is 


icerned with the process. 


The GUIDELINES FOR EVALUATING AN ESSENTIAL DRUGS PROGRAMME have been divided into two 
nm parts. One is called PROCEDURES FOR EVALUATION OF AN ACTION PROGRAMME ON ESSENTIAL 
IGS and the other is called MANUAL FOR CONDUCTING AN EVALUATION OF AN ACTION PROGRAMME ON 


SENTIAL DRUGS. 


The PROCEDURES consist of a number of questions which will be helpful in measuring 
ygramme effectiveness and efficiency. The PROCEDURES are quite comprehensive and 
‘ailed. This is intentional, because it has not been the idea to introduce a standard 
-mat for evaluation of an essential drugs programme, the reason being that the programme 

one country is often very different from the programme in the neighbouring country and 
is must be treated individually. It has therefore been decided to prepare the PROCEDURES 
a “cook-book" from which the "menu" can be chosen depending on the individual programme 
1 also on the emphasis of the evaluation. In newly established programmes the emphasis 
sht be more on the central functions of the programme whereas in more advanced programmes 


» emphasis might be more on the periphery, i-e., rational use of drugs. However, no 


‘dance can be given because it will always depend on the actual situation. It is 
he actual situation and decide 


srefore necessary, long before the evaluation, to assess t 
the right evaluation protocol. 


y steps involved in carrying out an evaluation from the 
en to the time the evaluation report is completed and 


The MANUAL discusses some of the necessary 
The MANUAL has been 


SHS / PHC/REVIEW/ 84/0, 


The MANUAL outlines the necessar 
ne the decision to: evaluate is tak 
scussed among high-level decisions-makers. 
>ps involved in fitting a country evaluation protocol together. 
»pted from the manual for conducting primary health care reviews, 


9, 1984. 
iduals who have a good knowledge of an 


INES is the indiv 
management or by members 


The target group for these GUIDEL 
y can be used by the programme 


sential drugs programme. Thus, the 
an independant evaluation mission. 
neva, November 1985 
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Executive summary: 


Summary bckground information on country and health care system 


PROCEDURES FOR EVALUATION OF 


AN ACTION PROGRAMME ON ESSENTIAL DRUGS (APED) 
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Amelysis of funecEtomuse . -» « » © Veule qe» @ 


6.1 


2 Financial management «+ +++ * * * 
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4 Health care delivery system .+-+ ++ * * * 


Budgeting, financing and costing +-++++*** * ° 
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6.2.2 Opergerama@l: aspects . "sg « = + © « «miams ° 

622.3 ASSemamnerCOBER +. « seugene, © » °° (ol eiae 


SEEMONIEOTiNG tem’ +s + is ikem 6 « + © eRe © ° 8 
S401 Indteseaeeina « 0 © sunineis « © « = «meme? © © Ce 
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Assessment of the input of the government and the donors. « « + 
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mumeesenay(S) seks s .< .« « @ gues 6 0 6 (© eo MME So 0d sue 
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Note; These procedures are in accordance with the method and 


glossary used in the "Health for All" series: 


no. 4 — Development of Indicators for Monitoring 
Progress towards Health for All by the Year 2000 


no. 5 - Managerial Process for Nati 
Development ational Health 


no. 6 - Health Programme Evaluation; and 


- no. 9 - the Glossary of Terms u i 
sed 
"Health for All" Series, nos.1-8 oe 


(1) 


(2) 
(3) 


(4) 


The number in brackets above and in the PROCEDURES refers to tle Fee 
references in sect 
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EXECUTIVE SUMMARY: OVERALL CONCLUSIONS AND RECOMMENDATIONS 


The most importa 

nt phase of the eval i : . 
4 E uation ex : 
-onclusions and recommendations. Braves consists ef etating overatt 


2) Overall conclusions 


On th i i 
> basis of the analysis of components and functions (see sections 4 and 5), a 
concise narrative evaluation statement shall be provided. 


b) Recommendations 


Most recommendations will be the result of the above-mentioned analysis. The 
recommendations must include information on who will be responsible for the necessary actio 
or approval and when it is required. ‘ ‘j 


1. SUMMARY BACKGROUND INFORMATION ON COUNTRY AND HEALTH CARE SYSTEM! 
1.1 GENERAL CHARACTERISTICS 


2 it-dis not necessary to include a general description of the country's geography, 
climate, ethnic and religious variations, social structure, transport and communication 
facilities etc., in the evaluation report. A short reference to relevant reports will be 


sufficient. 
1.2 POLITICAL/ADMINISTRATIVE SUBDIVISIONS 
Describe briefly and give a diagram. 


1.3 POPULATION 


Give the total population based on the last census and broken down into age groups and 
regions. If relevant, give figures on migration and refugees. 


1.4 NATIONAL INDICATORS 


State the rate of the following national indicators today and 10 years ago: crude birth 
rate, crude death rate, natural growth rate, infant mortality rate, neonatal mortality rate, 
mortality rate 1-4 years, maternal mortality rate, life expectancy at birth and literacy 


rate (male and female). 


1.5 HEALTH SYSTEM 


(MOH) organization in general with no particular 


Describe the Ministry of Health 
his will be done in section ee 


reference to the APED programme as t 


cen 
ld be obtained before the 


ystems shou ; ; 
years), it will be 


1 Information on country and health care s 
d out recently (last 2-3 


evaluation. If a PHC review has been carrie 
enough to make a reference to this review. 
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ROGRAMME 
2. BRIEF DESCRIPTION OF THE GENERAL FEATURES OF THE APED P 


———— e programme (formatio 
2.1 HISTORY: Dates of historical milestones 1n the tere rene ear aon ee 
of a drug committee, policy formulation, legislation, i oer oe and 6 ee 
(appraisal reports, plan of operation) were written, agop ple eee. the Se 
started to operate; when national and expatriate staff wer 


of reports). 
11 programme 


ich the objectives 


; vera 
2.2 PROGRAMME OBJECTIVES, INDICATORS AND TARGET GROUP: State the ove 
in receiver of the 


t to wh 
objectives, indicators (variables that help to assese the gts ea 
being obtained) and target group (which group in society wt 
programme). 

The strategy should in broad terms 
quantification of drugs, 


2.3 PROGRAMME STRATEGY: State the programme strategy. 
The detailed descripti 


describe how procurement, distribution, training, tae soD anc 
health education and policy and legal matters have been designed. 
can be found in section 5 (Assessment of efficiency). 


2.4 ORGANIZATIONAL LOCATION 


a) Give diagram and describe location within MOH, including formal and informal 3 
relationships with primary health care (PHC), Maternal and Child Health (MCH ) , Expande 
Programme on Immunization (EPI), Diarrhoeal Diseases Control (CDD), hospital and laborato 


services, health education, environmental sanitation, etc. 
b) Describe relationships with other ministries, including coordination and planning. 


c) Describe the relationship with nongovernmental health programmes (NGO's), their suppo 
and participation in the APED programme and possible conflicting areas. 


2.5 ORGANIZATIONAL STRUCTURE of APED (from central level to regions and peripheral units) 
Attach an organizational chart. 


2.6 MANAGEMENT STRUCTURE OF APED 


a) Committee: Does a steering committee (or other committee) handling APED matters exis 
If so, give date established, membership, and indicate dates of past meetings. 


b) Management: Who is in charge of the daily management. Attach job description as ant 


c) Expatriate(s): How many expatriates have been employed? If so, give titles and 
responsibilities and attach job descriptions as annex. Length of expatriate contracts? 


d) Describe lines of authority and res ibility fr Progr i 
: ponsibil om ogra i 
on y y gramme Director to regions an 


e) Describe lines of authority from higher level to Programme Director 
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2.7 PERSONNEL identified with APED 


a) Persons (excluding the Pro 


ramme Direct ; ee 
cantrel taser: 8 or) responsible for APED programme activities at 


How long | Special Responsibility Job description 
in post training 


b) Other persons with responsibilities for APED programme activities at regional and 
peripheral levels: 


acility (i.e., warehouse, Professional | Nature of Job description 
ealth centre, dispensary, category responsibilities 
raining school) 
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2.8 BUDGET 


a) APED Programme budget funds approved: 
in 1000 (local currency) 


Next year Oe 


19 
Curren Government Oth 


Government Other 


Past year 19 __ 
Government Other 


Category 


Purchase and distribution of 
- essential drugs and vaccines 


- essential equipment 
- dressings 


Salaries & allowances 


Supplies 


Travel & transport 


Rent 


Capital costs 


Other - specify 


Note: If a distinct budget for APED has not been established, estimates of the above 
amounts should be made. 


b) Non-financial support: list all APED programme assistance in the form of supplies 
(excluding essential drugs), personnel, etc. (including monetary equivalents where possit 


c) Commodity support: list the amount of essential drugs, etc., donated to the program 
and the monetary equivalents. 


d) Conversion rate: What is the conversion rate to the US dollar? 
2.9 GOVERNMENT AND DONOR INPUTS: describe with reference to: 

a) Personnel 

b) Supplies and equipment 

c) Training 

d) Funds 

e) Commodity support 


f) Other (specify). 
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2.10 HEALTH AND SUPPLY PERSONNEL INVOLVED IN THE APED PROGRAMME 


Re gion 
g Number and category of sanctioned posts,e.g.,no of 


physicians, nurses, storekeepers, etc. 


| > 


For occupied posts: 
rate per 1000 population 


2.11 FACILITIES WITHIN THE APED PROGRAMME 


Village 
health 
post 


Rural 
dispen- 
saries 


Urban 
dispen- 
saries 


Rural 
health 
centres 


Urban 
health 
centres 


Region 
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ROGRAMME 


APED P 
D SOCIOECONOMIC ASPECTS OF THE 
ee, a significant impact 


i ave had ; 
The economic and social environment of the country pai ate and social data sei 
he Essential Drugs Programme implementation. Some bast aaa Carne Programme. } 
eeoeath i he field will be helpful in evaluating the ec plan, Fanances Trade, 
be gathered in the the field are likely to be the Ministries Of | cee eer 
feabere ee, heeeare’ there may also be research Le atormaenens toca 
a bilateral aid organizations which are willing and able ee ean dand (aoe of 
an . d importing agencies may also have useful informa . cee ional ie 
eae A fina ae ee and of and trade statistics are the eas a coment. MepegEnn 
Sos an aaiksaued on a fequilar basis by the IMF; the ai ger neg coo aes 
the World Bank; Trade and Development coed ata aay 3 enes a nachenue een cig 
articularly useful sources for country~ Ceiigd “en haar 
Pat denmeion and information on governmen nd & 


expenditures. 


specifi i 
t budget allocations 4 


cribed below. Depending on 
t be possible to cover all t 
ude a qualitative 


The basic elements to be included in the analysis are des 
country circumstances and the availability of data, 1t may A Pe 
points. If this is the case, it would probably still be useful to anaes 
"best guess" appreciation of the situation, detailing the assumption . 


3.1 LEVEL OF DEVELOPMENT 


the total population, the gros 


Key indicators here are the rate of population growth, 
breakdown between urba 


national product per capita, literacy and school attendance rates, 
and rural population, infant mortality rate, life expectancy at birth, and any other 
relevant information such as whether the country is an oil producer, etc. A good source 
this data is the World Development Report. 


3.2 IMPORTATION OF DRUGS: Trade, balance of payments, foreign exchange situation and impo 
mechanism. 

In most countries, the majority of drugs or raw materials will have to be imported an 
the balance of trade and the availability of foreign exchange will have a significant eff 
on the success and continuation of the essential drugs program. Key indicators include t 
percentage of total imports devoted to drug (this can be estimated roughly if not precise 
calculated, by finding out total imports from IMF statistics and estimating total imports 
drugs, both public and private; it may be interesting to compare with such imports as 
gasoline and fuel, food, wood products, vehicles, etc.); the sources of the imported 
drugs; any restrictions affecting the import of drugs or raw materials, including the 
granting of import licences or foreign exchange licenses; exports of locally produced dru 
af any, and re-export of imported drugs. Exchange rates (official and semi-official) car 
found in the IMF statistics, but market rates must be gathered anecdotally in the field = 
used with See a or usually undervalue the currency, and governments are often 
sensitive about the subject. A qualitive appreciati i i 
eS ae pe q ppreciation of how serious the foreign exchang 


The role of foreign donors and aid agencies in alleviating the foreign exchange short 


may be significant; an estimate of how h.£ i 
would bé° useftl. : mich foreign exchange they supply for drug purchéz 


It is also useful to describe the mechanism of importation of drugs 
breakdown between public and private im i ; 
down ports, and is the same process 
description of the main organizations involved in drug jaeeteene cnetgl eee Gia 
; ing el) 


legal framework and status, their res i i 
pective volumes of imports i inci i 
whether they are subsidized by the government, etc, : Oe ice 


What is the 


If import duties are applied to dru 


gs what a : . 
types of drugs, etc. re the percentages applied to different 


39 ERICES 


How are prices calculated, in both the public and 
breakdown, with transport, profit margins, 
permitted to pharmacists? If possible, 


: d private sectors? What i 
ee oeaae costs, etc.? What act ar 
price level data should be gathered for 15-20 
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essential drugs, going back to the be 
dropped and by how much, 
Side by side? 


ginning of the program to show whether prices have 


What are comparable prices in the private sector, if it exists 


3.4 COST RECOVERY 


If a cost recovery mechanism was included as part of i 
describe it and comment on the success or lack rhekeee. eee che inte se 
the costs of drug supply, or only part of those costs? Was adequate provision nada oh 
expansion of the coverage and for increased demand for drugs, and for inflation? what” 
provision was made for indigent patients and chronic diseases? Did the Souccnsih ie 
personnel collect fees as expected? Could the scheme be replicated in other countries? How 
did prices for drugs correspond with people's willingness and ability to pay’? 


3.5 BUDGETARY ALLOCATIONS AND EXPENDITURES FOR HEALTH AND DRUGS 


This section deals with the place of the essential drugs programme in the health sector 
expenditures overall, the health sector's share of overall government expenditures, and 
allocations within regions, between urban and rural facilities and programs, and the role 
and importance of the private sector. 


Key questions are: 


a) What is the percentage of the overall government budget allocated to health? Include 
both Ministry of health and expenditures by other institutions, such as armed forces, 
education, national hospitals, semi-government procurement agencies, etc.--these may be 
significant and even exceed expenditures by the Ministry of Health. Show the trends over 
the past five years. 


b) What is the share of pharmaceuticals and vaccines in national health expenditures (MOH 
and others)? If possible, breakdown between essential and other drugs. 


c) What is the breakdown between recurrent and investment costs in the health budget? What 
share is allocated to personnel ? transportation? other significant categories? 


d) What type of investments are planned, and what kind of recurrent budget implications can 
they be expected to have? What will be the impact on the budget allocations for drugs? 


e) What is the breakdown between urban and rural programs? Preventive and curative? 


£) How important is the private sector? Estimate the expenditures on private medicine if 
possible. Include social security schemes, armed forces, and other major providers if 


applicable. 


g) What are the major sources of finance for the health system? (public revenue at central 


level, regional and local revenues, household expenditures, insurance schemes, ket 
non-governmental contributions, foreign aid and other external inflows. Describe briefly 


activities of major donors in health sector. 


Note: Several recent WHO documents provide explanatory getp and propose tabulation on 
specific aspects, reference (5), (6), (7), (8), (9), and ( ; 
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Ave ANALYSIS OF EFFICIENCY | 
ined in relatiol 


s obta : 
egui® tat10l 


The analysis of efficiency consists of a comparison of the rest oo eae oaplenen 
to the efforts made and resources used as well as the degree to whic 
complies with the planned implementation. 


: s are 
Before reviewing the efficiency of the programme, the following three step 


recommended: 


Step one 
t document 


Look separately at each component as it appears in the original projec 


(appraisal report or plan of operation). 


An APED programme is likely to have at least the following components: 


Selection of drugs, equipment and dressings 


Logistics 
ods, contract terms, locating and selecting 


4 <2.m procurement (procurement meth 
suppliers, procuring, monitoring supplies performance, quality assurance, 


local production of pharmaceuticals or import) 


Ff 
. 
oR 


A262 distribution (receiving and inspecting, storage and warehousing, inventory 
control, supply, delivery) 
43263 logistic support (maintenance and repair, environmental management of healt 


facilities, records and reporting) 
4.3 Manpower development 


4.3.1 formal training 

4.3.2 supervision 
4.4 Health education and information 
4.5 Policy and legal aspects 
Step two 


For each component state the immediate objectives, inputs, activities, outputs and 
assumptions and list the indicators and the portion scheduled for completion/provision by 
the time of the evaluation (see figure). 
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Figure 


Immediate 
ob jective! 


List aims 


ist indicators” 


List portion scheduled for 
completion(c)/provision(p) by 
the time of the evaluation 


Review progress 
ist assumptions® 


Immediate objectives: express the particular effect which the component is 
expected to achieve, if completed successfully and on time, e.g., "train selected health 
planning officers, medical officers, storekeepers and clerks in managing the drugs supply 
system with emphasis on procurement, storage, distribution, supervision and monitoring". 


4 Inputs: are money, personnel, materials, services, etc. to be provided by the 
benefitting country and/or WHO and other donors, e.g., ") technical expert to participate in 
drug supply management training or funds for a national level workshop on drug supply 


system, etc.". 


3 Activities: are the actions, the training and other substantive tasks to be 
carried out by the programme management, €.8-, ">? weeks of training of mid-level managers 
and drug supply operational staff in the drug supply system". 


4 Outputs: are the product of completed activities, e.g., "10 mid-level managers and 
25 operational staff trained in the drug supply system". 


5 Indicators: are variables that help to measure change, e.g., “an improved drug 

P . . . . . 
supply system, measured as the availability at any given time at each service delivery point 
of at least 90% of the essential drugs assigned to that level". 


6 Assumptions: is the term for external influences, factors, situations and 
conditions which are necessary for project success, but which are largely or completely 
beyond the control of programme management, e-8. "retention of trained personnel in 
functions for which they were trained". 


Step three 


Analyse efficiency by comparing actual with planned achievements and assess whether the 
result could have been obtained in a better and more economical way. In order to perform 
this analysis, a number of questions have been developed for each component. Some will be 
relevant for the actual situation, some not. The following sections 4.1-4.5 describe the 


questions that could be asked when comparing actual with planned achievements. 
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4,1 SELECTION 

4.1.1 SELECTION OF DRUGS 


. . 9 
a) Is there a national list of essential drugs: 


> 
ublic sector Or for both sectors: 


b) If yes, is this list only for the p 
c) Is it implemented? 
d) Are there lists by level of use? 


; : - 
e) If yes, how many items are on these lists? 


Ideal number of items 
Ideal nume_——————_ 


-- a 
Ref: Village health worker (VHW) 6-10 (see table 1--example f rom 


country) 
Dispennsary/health centre 30-40 (see table 2--example from a 
country) 
District hospital 60-100 
Consulting hospital 200 


£) Are all items on the WHO list of essential drugs?(11) How many are not? 


Do these items comply with the WHO guidelines for essential drugs?(11) 
pages 7-10 

ii) Can their inclusion be justified? 

iii) Who is determining these "exceptions"? 


Te not. i) 


g) Compare the actual lists with the original ones included in the first plan of action. 
Try to understand the why of deletions and additions (need, demand etc.) 


h) Are all items on the list really essential for that level of health care? 
i) Is the list complete? 
All the therapeutic groups should be considered, but inclusion of drugs on the list 


under study should depend on local conditions, specific prevalent diseases, capacities of 
the health workers involved and integration with other programmes (é.g., EPI, TB, Stes). 


Table 1 - example from a country 


LIST OF DRUGS FOR VHW: 


Analgesic 
Acetylsalicylic acid tab. 


300mg 


Anti-helmintic 
Piperazine tab. 500mg 


Anti-malarial 
Chloroquine tab. 150mg base 


Anti-anaemia 
Ferrous salt tab. 200mg 
Folic acid tab. 5mg 


Dermatological 
Benzyl Benzoate lotion 


Oral Rehydration Salt 


hthalmic 
Tetracycline eye ointment 1% 
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Table 2 - example from a country 


ARIES 
LIST OF ESSENTIAL DRUGS FOR RURAL HEALTH CENTRE S/DISPENS 


Ear drops 


Anaesthetic, local Chloramphenicol ear drops 5% 


Lidocaine 24 
Analgesics Oral Rehydration Salt 
anal gesics : ‘ BSS Ad iat 
Acetylsalicylic acid tab. 300mg 


Paracetamol tab. 500mg Ophthalmic 


Tetracycline eye ointment 1% 


Anti-allergic ee? 

Torphenami b. 4m xytoc i! 
Chlorphenamine ta g Se cance inate Seiad 
Anti-bacterial drugs 
- Penicillins 
Procaine Benzyl Penicillin 

inj. 4.8 mega U. + inj. water 


Plasma expander 
Dextran 70 I.V. infusion 


Triple Penicillin mixture Psychotherapeutics 
inj. + water for injection Ch lorpromazine tab. 25mg 
Phenoxymethyl Penicillin syrup Chlorpromazine inj. 50mg/2ml1 
250mg/ 5m1 


Phenoxymethyl Penicillin tab. 250mg Respiratory tract drugs 


Ephedrine tab. 30mg 


- Antibiotics, other Adrenaline inj. (1:1000) I1ml 
Tetracycline capsules 250mg heey 
Sulfadimidine tab. 500mg Vitamins 


Vitamin A tab. 
Anti-epileptic, sedatives 
Phenobarbital tab. 50mg 
Phenobarbital inj. 120mg/2ml 


Anti-helminthics 
Niclosamide tab. 500mg 
Piperazine tab. 500mg 

Mebendazole tab. 100 mg 


Anti-malarial/anti-protozoals 
Chloroquine tab. 150mg base 
Chloroquine syrup 50mg base/5ml 
Quinine inj. 300 mg/ml 
Metronidazole tab. 200mg 


Anti-anaemia 


Ferrous Sulphate tab. 200mg 
Folic Acid tab. 5mg 


Anti-schistosomal 
Metrifonate tab. 100mg 


Antacid 
Magnesium Trisilicate tab. 250mg 


Carthartic 
Senna tab. 7.5mg 


Dermatological preparations 
ential Violet crystals 
ydrocortisone 1% 

Benzyl Benzoate lotion 

Benzoic acid + salicylic acid 

(Whitfield's ointment) 
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-1.2 SELECTION OF ESSENTIAL EQUIPMENT AND DRESSINGS 


Is there a list of i : 
essential medica : : 
lapneweory mateciaie)t 1 materials (dressings, needles, equipment and 


If so, a comparison can be made with the following examples: 


a) Equipment list, Ministry of Health, K 
» Kenya, for Rural Health Faciliti 
b) List of dressing material for emergencies, Red Cross (Table a in 


Table 3 


List of essential equipment items needed with the essential drugs in rural health facilities 


Equipment item UNIPAC code Required 
i . HC/HSC DISP 
Bottles Albustix urine tests 0921100 1 - 
Luer syringes 2 cc. nylon 0785670 40 20 
Luer syringes 10 cc. (autoclavable) 
- not disposable nylon 07856784 10 2 
Hypodermic needles Luer s/s 0750500 10 - 
(children) 

Hypodermic needles Luer s/s 0749500 50 25 
Clinical thermometres F 0481050 10 5 
Auroscope 0660000 1 1 
Sphygmomanometer aneroid 0683000 2 1 
Stethoscope 0686000 4 1 
Test tube racks 0968518 1 - 
Test tubes 0978995 20 - 
Spirit burner 0515500 1 - 
Test tube holder 0930000 1 = 

onocular microscope 0960000 1 - 
Slides 0969000 1 = 
Slide covers 0934001 1 - 


Source: Government of Kenya, Ministry of Helth, New Management Systems of Drug Supplies to 
Rural Health Facilities, Training Manual, May 1982. 


1 parcel containing: 
rolls gauze bandages 6 cm x 10 m 
rolls gauze bandages 8 cm x 10 m 
elastic bandages 8 cm 
elastic bandages 10 cm 
sterilized gauze pads 10 x 10 cm 
sterilized gauze pads 10 x 20 cm 
sterile covers sponges 7,5 x 10 cm 
absorbent cotton wool 
triangular bandages 
adhesive bandage 5 m x 6 cm 
dhesive bandage 5 m x 8 cm 
rolls adhesive tapes 2,5 cm x 10 m 
rolls adhesive tapes 5 cm x 10 m 
skin desinfectant 
scissors + 1 forceps 


1 
2 
2 
1 
2 
1 
5 
1 
5 
1 
1 
1 
1 
1 
1 


I ; he Red Cros Medical Division tandard Emergency 
n ernational Committee of t R 8, S 
bre Geneva 1983. Price appr- 200 S.fr. (84) covers 50-100 persons in an 

, 


emergency situation. 
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.2 LOGISTICS 
‘ gistics 1s defined as the process 


: : ra uhe 8) 
This part of the evaiuation deals with LOGISTICS 


i i ini i supp Lies. 
ng, maintaining and transporting 
— number of questions should be 


alee dng eference (12) 


‘athens 
When evaluating the performance of the pe Hise has been made of r 


i i ive 
raised. In formulating relevant questions, extens 


and (17). 


4u2.4 PROC UREMENT 
it involves i) choosing of 


Beacarenenk ieameeenroces® OR Maer) ae: iii) locating and selection of 
ij ifi i trac 
procurement method, ii) specification of con and v) quality assurance of drugs. 


suppliers, iv) monitoring supplier performance, 


A. PROCUREMENT METHOD 
bein 

a) Which of the following four (or combination of the four) procurement methods are gZ 
used in procuring drugs, equipment and dressings: 

1 open tender (public and unrestricted bid) - a 

; 3 - 

2 restricted tender (suppliers must be registered and app 

3 negotiated procurement (suppliers are well known and can be ap proached oe ae 

4 direct procurement (suppliers' wholesale or retail prices are used directly). 


d in advance) 


Do the methods used ensure the availability of safe and effective drugs of acceptable 
quality at the lowest possible cost? Are the obtained prices in accordance with prices in 


the commercial market? Compare with UNICEF prices (17). 


b) Are drugs labelled in accordance with standardized labels of APED programmes, e€.g., 
generic name, dosage strength, presentation, quantity contained, manufacturing date 
including expiry date, batch number, any special usage and storage precautions and name anc 
address of manufacturer, 


B. CONTRACT TERMS 


Analyse existing contracts and evaluate whether they maximize the likelihood of 
satisfactory supplier performance, without leading suppliers to raise prices? The contract 
should at least have provision for the following: 


™ trade terms 

= price and currency 

= payment terms 

~ quality standards 

- nomenclature and labelling 
= product specification 
= financial guarantees 
= delivery date 

= patent provisions 

= packaging 

= expiry date. 


C. LOCATING AND SELECTING SUPPLIERS 
LER 


a) Describe the supply sources (including percentage of each 
and dressings, €-8-, government production, local private manu 
manufacturing, donors, international procurement services, 


of the suppliers. An example of a system to evaluate new s 
reference (12), page 154, 


Source) for drugs, equipment 
facturing, foreign 

etc. and analyse the reliabilit 
uppliers can be found in | 


ppliers is Sufficient. Are ne 
ement, international procureme 


informally, by test 
pages 6-9)3 Mf purchases | 
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) 


Analyse whether new su i 
ppliers are ch : . ; 
rformance, delivery date, etc. oosen according to, e.g., price, quality, past 


D. PROCURING 


Analyse the system of i 
procuring and che : : 
positive: & ck whether the answer to the questions are 


a) Is there a standardized supplies list? 

b) Are there sufficient staff adequately trained? 

c) Are there sufficient forms and records? 

d) Are donations made in accordance with standardized supplies list? 
e) Is there a defined supply period? 
f) Are stock records up to date? 

g) Have re-order levels for major stock items been identified? 

h) Are reserve stocks maintained? 


E. MONITORING SUPPLIER PERFORMANCE 


Does there exist a formal monitoring system for evaluating supplier performance? If so, 
does it provide the following information? 


a) recording of supplier participation, 
b) response to inquiries, 

c) delivery time, 

d) adherence to delivery instructions, 
e) provision of documents, 

f) packing and labeling, 

g) cost improvement suggestions. 


More details can be found in reference (12), page 157. 


F. QUALITY ASSURANCE 


The purpose of quality assurance in an essential drugs programme is to make sure that 
each drug reaching the patient is safe, effective and acceptable. Analyse whether the 
quality control system ensures safe, effective and acceptable drugs. 


A well functioning Quality Assurance System should be based on stated policies and 
procedures aimed at assuring drug quality; access to a drug quality control laboratory, well 
trained and adequately supervised quality control inspectors; procurement of drugs in 
accordance with Good Manufacturing Practices (GMP) (14) and international pharmacopoe1as and 
participation in the WHO Certification Scheme on the Quality of Pharmaceutical Products 


Moving in International Commerce (14). 


When evaluating the Quality Assurance System the following questions might be helpful: 
gs from supplier to end 


d to assess and maintain the quality of dru ) 
a) What methods are used to <a: ae 


user (patient)? Is there a plan for taking samples of the drugs? 


realistic? 


en an issue in the programme? In what way? What effect 


b) Has bad quality of drugs ever be ey health personnel? 


has this had on the public health? On the programme? 


c) To what extent is physical inspection made of the drugs Siopee Meg? in the central medical 
store? Are the inspectors well-trained and adequately supervisecd: 


. ; 
d) Are laboratory analyses ever used to assess drug poe age “ta aie etle ar 
samples chosen? Who performs the analyses? Is the testing laboratory 
what procedures are used to inspect the 


e) If drugs are received from suppliers in kits, 


kits? 
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plier to dispensary? Can drugs 


i i om su : J Zand 
f) What are the conditions during tr enone ae ee Of ance thas at io 
i hese conditions? a 
seriously affected by tt 4 : 
y at port-of-entry, ii) at centra 


5 ob ce ore luated i) s and 
e facilities periodically eva : alth centre tg 
OE acme oes f i d sonal warehouses, iv) at heait dee eeainieeeee 
iii) at district and reg o is responsible acdad 


war ehouse 
If so, what are the procedures and wh 


dispensaries? 
they have? d other personnel a 
icians an 
‘llage health workers, nurses, physicians @ ea? 
h) Is there a system for v1 g lity? If so, which sys ? 


the peripheral level to report problems with drug qua 


G LOCAL PRODUCTION OF PHARMACEUTICALS OR IMPORT 
e eee 


: ; ili ing: 

Describe the local pharmaceutical production capability doing 
. i ici neces and other 
- primary manufacturing (the production of active See, oe 

ancillary substances used in pharmaceutical fore eee ae pet iay \toseeae 
z secondary manufacturing (the processing of medicinal subs a corel 
with ancillary substances, to produce the required Se coucianaa eee a 

i i s in manners appro 

packaging (the packaging of dosage form ann Be en navertiit aad cou 
the preparations and their intended use, conditions ’ 


a) 


conditions and time of storage). 


. hd 9 
and assess the potential for the future (Does government policy promote local production? ) 


b) Are locally-produced drugs essential? 


c) Assess whether locally-produced drugs are competitive with similar drugs on the 
international market? 


a.252 DISTRIBUTION 


Distribution deals with all of the activities required to receive drugs from the 
suppliers and to move them safely, securely, and expeditiously to the many points in the 
health care system. 


H. RECEIVING AND INSPECTING 


Receiving 


Evaluate whether receiving procedures are based on: 


a) Goods received are the same as goods ordered 

b) Adequate storage and holding facilities at the point of entry 
c) One person responsible for receiving supplies 

d) Customs clearance work smoothly 

e) Adequate warning of arrival of drugs 


Inspecting 


Analyse whether inspecting procedures are based on; 


f£) Established system for inspection of supplies 

g) One person responsible 

h) Sufficient staff to do the work 

i) Adequate documents and forms for inspection 

j) Action taken and results recorded when discrepancies are found 
k) Unauthorized substitutions of goods rejected 

1) Price charged compared with price quoted 

m) Physical inspection carried out on 
n) Adequacy of packing checked 

o) Check for quality of received supplies 


all incoming goods 
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- STORAGE AND WAREHOUSING 


Each store should be assessed in relation to: 


) Physical arrangement of stores (Central Medical Store and regional/district stores) 


Structure: 


- Is the building structurally sound? 
- Is there adequate light? 

- Is there adequate cross-ventilation? 
- Is the floor strong and even? 

- Is it vermin, bird and insect proof? 


Security: 


- Are all doors lockable? 

- Are roof and air vents secure? 

- Are all windows barred? 

- Is there a control system for visitors? 

- Have there been no significant losses from the store? 


Services: 


- Is there suitable artificial lighting? 

- Is there drainage? 

- Is there fire-fighting equipment and is it regularly checked? 

- Do staff know how to use it? 

-~ Is there an alternative supply of electricity for cold rooms and refrigerators? 


Capacity: 


- Is the store big enough? 
- Are there special stores for toxic/inflammable goods separate from the main building? 


b) Handling facilities 


- Is there equipment to move stores around the warehouse? 
- Is there protective clothing for porters, €.8-, for those working in -20 C cold rooms? 
- Are there enough shelving racks and cupboards to hold your stores? 


c) Suitable staff 


Are there suitable staff for the following: 


- Management and control of store? 

- Receiving and issuing supp lies? 

- Handling supplies? 

-~ Maintaining stock control? 

- Maintaining building, grounds and sa 
- Maintaining sanitary facilities? 

-~ Maintaining specialist storage space, 
- Maintaining cleaning schedules? 


fety equipment? 


e.g., refrigerators and freezers? 


have a job description? 


- Does each category on? — 
n-the-job training and retraining? 


- Are there facilities for o 


d) Standard list of stock items 
Is there a specific description for each stock item? 
- Are all stock items included on the list? 


i d? 
the list regularly update : 
- as the list distributed to sub-stores and units? 


- Do received orders use the same wording to descri 


be items as those in the list? 
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e) Special storage 
2 
d s kept in a double locked store* — 
Son : disinfectant, kept in a separ 


? * 
parate store: Me i sie 
d freezers, Or cold rooms 


- Are ate store? 


- Are contaminated supplies, CeB+s aie 
- Are inflammable goods kept 1m a special 
- Are there sufficient and suitable refrigerators an 


ere ° 2 - 
and other heat-sensitive supplies! — ‘cal equipm 
- Are there secure stores for small items such as dental and surgic qu1p 


ent? 


f) Supervision 


- Is there a supervisor? pee: 
- Is there a schedule for supervisory visits? 


. e » 
- Is there a supervisor's checklist? oes 
- Is there regular temperature recording for coldroom, refrigerators and freeze 


- Are drugs and supplies physically checked regularly? 
Are staff encouraged to make suggestions? 


g) Wastage 


drugs which do not reach the intended 


- Wh is th astage rate for drugs, i.e. frat 
By +8: none g ; ii) physical damage, iii) damage 


population? Causes of wastage could be i) drugs expired, 
by climate (sun, heat, humidity), iv) pilferage, and v) losses. 


- What is the procedure for expired drugs? Are they being consumed or are they being 


returned? 


J. INVENTORY CONTROL 


Analyse the inventory control system and check whether the answer to the following 
questions are positive: 


a) Recording of inventory 


- Is the information recorded the same as the quantity of stock in store? 
- Is there a schedule for resupply? 

- Are inventory reports regularly received from all sub-units? 

- Are inventory reports from sub-units regularly checked? 

- Are there sufficient quantities of all inventory forms? 

- Are there sufficient staff? 

- Are recording forms regularly reviewed and brought up to date? 


b) Sufficient stock level 


- Are there reserve stocks sufficient to meet reasonable demand? 

- Have all requests been supplied in the last year? 

- Are all supplies within their expiry date? 

- Are predetermined maximum stock levels set for most items? 

- Are all items below the maximum stock level? 

- Are allowances made for seasonal demand? 

- Is stock rotated on a "first in, first out" basis? 

- Is there a defined system for resupply? 

~ Is there a policy for disposal of damaged or expired supplies? 

- _ as cost aa‘ supplies well-balanced, e.g., not too frequent supply? 
- Is there an effective means of communicati i i ; 
Me chere ree etic iaiece tion between each link in the supply chain? 


c) Inventory control at health centres/dispensaries 


~ Is there a regular storekeeper? Has he/she been trained? Who is supervising? 


- How often are the health centres/dispensaries out of stock? 
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K. SUPPLY 


Analyse the supply system and ch 
posi kive: ¥ check whether the answers to the following questions are 


Documents used for ordering supplies 
- Do forms have all necessary information? 
- Do staff understand forms? 

- Are forms completed correctly? 

- Are there any unnecessary delays? 


Supplies at sub-stores 


- Are there clear supply procedures? 

- Are there established supply intervals? 

- Are quantities compared with consumption? 

- Are quantities compared with morbidity? 

- Are supplies within their expiry dates? 

- Are heat-sensitive supplies kept cold? 

- Do all sub-stores have stocks of all essential supplies? 


L. DELIVERY 


Evaluate the delivery system and check whether the answers to the following questions are 
positive: 


Distribution system 


- Are there adequate sub-stores? 

Are there established delivery policies? 

-~ Is the time taken to travel to each sub-unit known? 

Are the shortest routes used? 

- Are there contingency plans to cope with breakdowns in the normal distribution system 


Transport plan 


-~ Have alternative uses of transport been assessed in establishing the preferred transport 


policy (e.g., rail, air, road haulers, other ministries' transport)? 
- Has the most economical and practical mode of transport been chosen? 


- Are vehicles' needs automatically received for additions and replacement regularly? 
-~ Is there a transport manager? 


Transport facilities 


- Are there sufficient working vehicles to meet your needs? 
- Are there alternative types of vehicles to meet different needs? 


-~ Is there a limited number of makes in use? 


- Are there adequate supplies of fuel? 
- Are there sufficient numbers of trained drivers? i: , 
ee < na — 
- Is there specialized transport for heat-sensitive supp*1e 
- Are vehicles used to their best advantage and not limited to a single part of the supply 


system? 
4.2.3 LOGISTIC SUPPORT 


M. MAINTENANCE AND REPAIR 


Assess the maintenance and repair system and check whether the answers to the following 


questions are positive: 
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Maintenance and repair of vehicles 
Maintenance ane *e 


° ° 9 
s for routine maintenance: 


ts? 
d mechanics? 


- Are there maintenance schedule 
- Are there adequate stocks of spare par 
there sufficient numbers of traine 


- Are 
° ° . 9 p ° 
- there job specifications for all staff? : s can be maintained 
- a there re equipped and accessible workshops where vehicle 
9 
eect hicles may be kept under cover? 


- Are there suitable garages where ve 


- Are there adequate safety facilitie 
- Are there repair and maintenance manuals and spare par 


- Are there set procedures for vehicle repair? 
- Do drivers keep vehicle logs? 
- Do speedometers work? 

- Are vehicle mileages logged? 
- Are there adequate stocks of: 


bulbs and tyres? 
- Are there adequate stocks of forms and records? 


s and fire appliances? 


ts for all categories of vehicles 


; i Z 
spare parts? lubricant? consumables, e-8-, oil filters? 1 


Maintenance and repair of equipment 
. . * bd Ld Ld 9 
- Are there maintenance schedules for routine maintenance of major items of equipment? 


- Are refrigerator user guides in use? 

- Are there sufficient numbers of trained repairmen? 

- Are there job specifications for repairmen? 

- Are there sufficient equipped and accessible workshops? 

- Is there an effective equipment failure report system? 

- Is there an equipment status report for major items? 

- Are maintenance and repair manuals and spare parts catalogues available? 

- Are clinical staff trained in the routine use, care and maintenance of equipment? 

- Are there adequate stocks of: spare parts for refrigerators? spare parts for freezers? 
fuel (gas or kerosene)? spare parts for dental equipment? spare parts for other equipment 
- Are there adequate stocks of forms and records? 


Maintenance and repair of buildings 


- Is there any routine maintenance of buidings? 

- Is the works department responsible for maintenance of stores adequately equipped? 
- Is there regular cleaning and inspection of stores? 

- Is there an effective system for reporting damage to buildings? 

- Are the materials necessary for repair and maintenance available? 


N. ENVIRONMENTAL MANAGEMENT OF HEALTH FACILITIES 
OE cain lll aa cs Anchen 


Analyse environmental management of health faciliti 
acilities and check wheth 
following questions are positive: a 


Standard of cleanliness, hygiene and safety at stores and health facilities 


Cleaning: 


- Is there an adequate supply of safe water? 

- 5 a facility clean inside and outside? 

- Is there a person responsible i ili 
[Sere ae for keeping the facility clean? 
- Are there enough cleaners? 

- Is there sufficient equipment and cleaning materials? 

- Is there adequate refuse disposal? 
~ Are there specific measures for dis 


Hygiene; 


? 
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Do staff use special clothing to wear whilst in these areas? 


Are the sanitary facilities sati 
isfactory, e.g., wash basi ? 
Is the drainage system satisfactory? e a 


Are there hygiene notices posted, e.g., for handwashing? 


afety: 


Are there regulations for handling inflammable, dangerous or toxic substances? 

Is there protective clothing for those who require it, e.g., dust masks, g0 les te? 
Is there an adequate level of lighting and ventilation? thesis 
Are there working fire-fighting appliances? 

Are they regularly inspected? 

Do staff know what to do in case of fire? 

Is one person designated as fire officer? 

Is there a working fire alarm? 

Are there adequate safety precautions in laboratories, e.g., fume cabinets? 

Are there first aid kits in stores and high risk areas? 

Is smoking restricted to specific areas? 


. RECORDS AND REPORTING 


valuate the records and reporting system and check whether the answers to the following 
uestions are positive: 


eporting between the periphery and mid-level management 


Are stock consumption reports routinely received each month from all peripheral units? 
Are comparisons made between stock consumption and use, e.g., doses of vaccine 
ssued/number of doses of vaccine given/number of fillings made/quantity of amalgam used? 
Is morbidity data available? 

Are there established treatment schedules for essential drugs? 

Are comparisons made between stock consumption nd morbidity data? 

Is all reported data necessary? 

Is all essential data reported? 

Is there routine feedback to the units at the periphery? 
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4.3 MANPOWER DEVELOPMENT 


4.3.1 FORMAL TRAINING 


mainly: 


A. 


j i in view 
carried out with several objectives in 


Training for essential drugs programmes 1s 


i ial drug supply 
to increase numbers of personnel with skills related to an essentl g 
portance of an essential drug policy for 


management system; 
= to promote increased aver ee Ee. 
atus of the popula ; ; : is 
- Sn canaineeee the inter-relationsh1p a he coer ee Be 
essential drug management system: selection, ee emes , & 
distribution, monitoring and evaluation, and drug use; a 

= to improve skills of health facility personnel in respec 


prescribing practices. 


f the im 


better diagnoses anc 


TRAINING INFRASTRUCTURE 


Evaluate the training infrastructure and check whether the answer to the following 


questions are positive: 


B. 


Is there a single appointed focal point (training officer or preening unit) 
responsible for organization and coordination of training activities: 


= Has a formal/informal assessment been made of manpower training needs throughout 
the APED programme including the balance/maldistribution of trained personnel? 


= Have formal/informal projections been made of training needs for the next l - 5 
years? 


a Are premises for different training purposes available and accessible? 
NUMBERS, ACTIVITIES AND BUDGET 


For both a structured training programme (on-going at regular intervals) and ad hoc 


training activities, compare the following questions with the Plan of Action and note 
discrepancies: 


- How many trainees have completed training activities in the past 3-year period? 
= Types of training activities? 

= Numbers trained in-country, nearby country, abroad? 

- Has the yearly budget allocation for training purposes been sufficient? 
TRAINING MATERIALS 

Compare the following questions with the Plan of Action and note discrepancies: 

Have specific training materials been developed? 


Have training materials been developed for health 


: : facili : 
oudgagnostic procedures and pres@eibing drugs? 1lity personnel at different lev: 


Have training manuals for trainers been developed specifically? 


Have teaching aids - posters, slides, wall charts, etc 
Y . 


levels of health care facility? ~ been developed for differen 


Have teaching/education materials been dev 1B ? 
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- SELECTION OF TRAINEES 


In the table, 


you will find the target groups which should have been trained by the 
rogramme. 


IDENTIFICATION OF TARGET GROUPS TRAINED AND PLACE OF WORK 
LACE OF WORK 


Place of Work 


Sess ER EE ee eee 


ersonnel/Function Port of Central District Hospital Health Dispen- Drug* 
entry level level Centre sary Control 


7 


Pharmacy Drug University/ 

store Teaching 
Authority outlet Institu- 
tion 


- Policy makers/Health 
Administrators/Managers x xX 


- Procurement officers xX 


- Drug distribution and xX xX x xX xX xX 
Logistics personnel 
(Pharmacists, administra- 
tors/clerical and storage 
personnel 


- Pharmacists/dispensers/ 
assistants 


a. general pharmacy/ x X X x X 
dispensary 


b. regulatory control Xx 


activities 


. Medical Officers/Nurses xX xX 4 
auxilliaries/Prescribers 


. Pharmacy/Medical Students/ 
other health personnel 
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4.3.2 SUPERVISION 
A passive definition 


VISION. : 
ee reporting, 


1 procedures (records, ¢ 
o consider supervision as an 
Supervision 1S probably a 


There is a number of ways of defining 
would be to consider supervision as contro 
etc.), and a more active definition would be c 
educational tool, i.e., one-the-job training. 
combination of both definitions. 


drug 
health care system and the 
pe em is supervised by the 


d by the logistic 


In relation to an APED programme, me 
supply system must be supervised. The health care sy ar 
medical staff, whereas the drug supply system 1S super 


staff. 


Medical supervision 
nee... ee 


Medical supervision is normally given by a superior (doctor) to : 
subordinate (para-medical assistant), but can just as well be given by a = 
trainer, a fellow para-medical assistant who has more knowledge, or an pes e 
person who knows how to convey the message, ©-8-; a teacher with knowledge 
about sanitation, community organization, public health, etc. To evaluate the 
medical supervision, the following key questions should be investigated: 


a) who is supervising whom? 

b) how often does supervision take place? 

c) does a regular schedule for supervision exist? 

d) is there a system of follow-up to see that observations are implemented? 
e) do the supervisors have the tendency to "take over''? 

f) have the supervisors been trained in how to supervise? 

g) is there transport available for supervision? 


Logistic supervision 


Logistic supervision is given to the periphery by the middle management 
and to the middle management by the senior management. Supervision should not 
be a system where a senior group checks on a more junior group, but rather a 
System to make sure that the communications system works to and from the 
periphery. To evaluate the logistic supervision, the following key questions 
should be investigated: 


- are there staff identified as supervisors? 

- do they have and use a regular schedule for supervision? 

- are there checklists available to help them? 

- 1s there a system of follow-up to see that observations are impelemented? 
218 there any static supervision (analysis of routine reports)? 

aes hmm information down to the units as well as collecting it 
- is there transport available for supervision? 
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-4 HEALTH EDUCATION AND INFORMATION 


Hes F ‘ d ; 
ealth education and information is carried out with the objective of: 


PDPateEs that all health personnel in the programme area are aware of the 
programme, its purpose with regard to PHC and the kind of support they will b 
given to cope with the new situation caused by the programme 5 i 


- ensuring that local opinion leaders and other elites know about the programme and 
1ts purposes and become motivated to play an active supportive role 


= enabling health personnel and opinion leaders to inform the general public about 
the programme and its importance to improve PHC. 


.. HEALTH EDUCATION AND INFORMATION AT THE NATIONAL LEVEL 


) Is there a national health education service? If so, is this unit involved in the APED 
yrogramme? 


.) Is the health education service or another service involved in ensuring that the policy 
ind principles of essential drugs are included in the curriculum of: 

= medical students 

oo nursing students 

= other health personnel 

= school teachers 

= public information personnel 

= workers in other sectors whose functions involve education for health. 


-) Describe briefly the information about essential drugs at the national level since the 
start of the programme specifying medium, subject matter, target population, frequency and 
those involved. 


1) Is there an identifiable amount budgeted at the national level for health education 
sbout essential drugs? If so, how much? 


3. HEALTH EDUCATION AND INFORMATION AT THE REGIONAL/DISTRICT LEVEL? 
1) Is there a health education service in the region/district? 
») What are the means used for informing people of health problems? 


; HEALTH CENTRE/ DISPENSARY LEVEL 


ment officers, health personnel and 


3) At the introduction of the programme were govern and — 
ts rationale and the kind of activities 


local opinion leaders briefed about the programme, i 
-o be carried out? 


P Baath , , ; ; , 9 
») Were the health per sonnel trained in improving their communication skills? 


>) What kind of education and information material has been produced and introduced 


= hand-outs 

~ posters 

- booklets on the drug policy other 

= flip-charts for training 

- tape-recordings about drug behaviour 
- street announcements 

* song and local dramas 

- advertisement in papers 

- radio programmes. 


2 
1) Has the logo of the programme been introduced? Tf so, to which Segres! 
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and information about essential drugs in 
subject matter, target 


e) Describe in some detail the health education 
NGO, health personnel, 


last 6 months specifying medium (play, demonstration, etc.)s 
population, frequency and those involved (education department, 
cultural groups, opinion leaders, etc )? 

£) Does the health education and information support the programme in relation to: 


e and ensuring saf 
instead o 


both in reporting illegal sal ety while in the health u 
- the public accepting advice from the health workers, f£ insisting on drugs? 
~ unnecessary use of hospital facilities? 
- the puble not insisting on injections i 
- the public refraining from pressuring rura 


drugs? 


- drug security, 


to be taken orally? 


l health staff into prescribing unnecessary 


nstead of drugs 
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-5 POLICY AND LEGAL ASPECTS 


9.) POLICY: 


The national i i i : 
BE a Sar pee ere hone “es or another an essential drugs 
ie edi ve tion Geile Be : ; ave to be as sessed. It will also give a 
gree o committment of the government regarding the APE 
implementation of the concept of essential drugs. ss “as D°and the 
- ee there a national drug policy? 
ee ec ok of this policy (health goals, economic goals, 
=~ In which aspects can this policy influence the APED (positive or negative)? 
= Is there a national list of essential drugs? : 
= If yes, is it adopted (for the public sector i : 
= What is the Be iiey toward the _filieap ae OF ORGAO # ire? 
= In which ways does this policy influence the effectiveness of the APED? 
= Is the government fully committed to the objective of the APED? Analyse other 
aspects of the drug policy im order to see if the global policy of the government 
is in accordance with the APED. 
= Is there a National Drug Committee set up in order to draw list of drugs or to 
revise the existing lists? 
= How many drugs are on the market? 
= What criteria are used for registration of drugs; are they similar to the ones used 
for selecting essential drugs (11)? 
= How does the parallel system influence the APED? 


4.5.2 LEGAL ASPECTS 


Drug legislation and regulatory control systems should assure that safe and effective 
drugs of good quality and for the intended uses reach the consumer. 


A. THE BASIC ELEMENTS OF THE DRUG LEGISLATION 


This section is dealing with most of the elements on a national drug legislation. 
During an evaluation all these aspects have do not have to be assessed. The first question 


should be: 


_ Is there any distinction in the legislation between essential drugs and other drugs? 

= If yes, indicate in the following questions how the essential drugs are treated 
differently from other drugs. 

— If no, try to assess the elements in the legislation which are relevant to the APED 


programme. The following questions might be useful: 


a) General 


= What are the present laws referring to drugs? 
= Is there a Drug Act? 
- What basic elements are included in this Drug Act? 
- Is the essential drug list endorsed by law? 
b) Control of import of drugs 
= What are the conditions in the legislation for importation of drugs? 
- Is a licence required: : * : 
- for imports of drugs in manufacturer's original packaging, 
- for imports of bulk drugs by the manuracturer, | 
= for unregistered drugs on a case-by-case basis: 
- Who is qualified to import drugs? 
= Which type of licences? 
c) Control of export 


the conditions for export of drugs? 
e required for export? 
d-keeping procedures wi 
o export drugs? 


= What are 
o Is a certificat 


a Are there recor 
Who is qualified t 


th respect to exported drugs? 
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s 
d) Control of manufacturer , 
| an enterprise may D 


d drug J 
d of formulate oneneeee ek cauaae 


i i ifying the kin 
is Is a licence required specifying eG. of. i eeeeD pe 


allowed to manufacture based on the resu . : 
So eta ee ee ttow Good Practices in Manufacture an 
; ae A ie 
= To get the licence, 1s 1¢ require 


2 
Quality Control of Drugs (industry level) (18)? 


e) Control of distribution 
of drugs which could b 


. ic distribution 
tends to avoid a chaotic facilibiemae all leve 


: i i is area 
Legislation in th per storage 


prejudicial to the patient and ensures provision of ie 
and effective supervision by responsible technical staff. 


for distribution of drugs? 


rz nditions in the regulations | 
a owed treatment establishments, pharmacies, 


- Are the allowed channels of distribution ( 
drug warehouses, etc.) fixed by the law? 


f) Drug registration aimed at controlling drug marketing 


Has an inventory of the drugs on the market been conducted? 


= Has the number of pharmaceutical products been reduced? > 
What kind of information is needed by the Drug Control Administration before the 


registration of a new drug? 
= Is the WHO Certification Scheme for Products moving in International Commerce us 


(14)? 


g) Labelling 


= Are there national specifications for labelling? 
= Are there provisions for control: 


- of the container? 
= of the outer package? 
- of the package insert? 


For example, does package labelling require the generic name, dosage strength, 
presentation, quantity contained, manufacturing date including the formula, name 
and address of manufacturer and product expiry date, batch number, any special 
usage and storage precautions, name and registration number? 


h) Information and advertising 


- Are there regulations on how and where it is permitted to advertise drugs? 
“> What are these regulations? 
> Is advertising prohibited in certain circumstances? 


i) Scheduling 


= Is there an official published national formulary? 
= Is there an essential drug list enforced by the law? 


¢ 


j) Price control 


- Are there provisions in the legislation for the control of 


; = 
available on the market (maximum DELTGES 4...) 7 prices for drugs 


B. THE DRUG CONTROL ADMINISTRATION 
C—O MON 


personnel, 
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The Drug Control Administration is of 


prime importance because it is the body that 
issues the various authorizations granted, 


whatever the stage of the drug. 


In this area, different countries have created different institutional mechanisms and 
different authorities in order to give effect to the legislative scheme. It seems that 
there are no standard models and the evaluation will have to be adapted to the specific 
country. However, in some circumstances in order to suggest ways to improve the 
functionning of the APED, the following questions could be asked: 


a) Is there a drug control administration? 

b) In which ways does this administration influence the APED? 

c) Which functions of the drug control administration are directly related to the APED? 

d) Describe these functions and evaluate their efficiency (registration, inspection, 
dissemination of information, etc.) 


a 
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5 ASSESSMENT OF EFFECTIVENESS 
. Ae Ee | 
her the results obtained a1 


ss consists of an analysis of whet 


Assessment of effectivene 
ramme ; namely: 


in accordance with the objectives of the APED prog 
y health care level of effective, affordable essenti ie a@ 
that at least 20 essential drugs will be avaiia 


lk or travel 


- regular supply at the primar 
and vaccines; specifically, 
the population within one hour's wa 

; ahs ibili and proper us‘ 

- improvement of the health situation through availability, accessibility prop 
of drugs and vaccines. 

: i i in a sho: 
The evaluation proposed below is not very detailed but it can be See cece ee 

period of time at the level of rural health facilities. To assess ane ais oO 

and impact will require much more detailed work at the community level sf <a 

workers, community leaders, home level), in health centres, depen ae a ; 
hospitals. It will also involve work at the regional and national level. 


5.1 AVAILABILITY OF DRUGS 


the best way to assess the availability of drugs is by doing . 
easure the actual stock of essential drugs and by 
ff in these health facilities on the actual state 


At the periphery level, 
survey in a few health facilities to m 
performing interviews of the medical sta 
supply of essential drugs. 


A. SURVEY 


Objective: Analyse a representative sample of health facilities which serve under th 
APED programme in order to assess the availability of essential drugs. 


Therefore the best way to choose the number of health facilities to be surveyed durin 
the evaluation would probably be the following: 


1. select the administrative units covered by the evaluation, 


2. stratify the administrative units according to the time the units have been part of t 
programme, 


3. divide each strata into smaller geographical units (clusters), 
4. decide how many clusters can be covered time wise during the mission, 
5. divide the number of clusters by the number of stratas, 


6. if the number of clusters within each strata is roughly the same, select randomly as 

many clusters from each strata as the result given in point 5. If the number : 

same, select number of clusters in proportion to strata size oS ee 
> 


re if it will not be possible time wise to visit all h 
stratify the material into, e.g. hospitals, 
as described in points 4, 5 and 6. 


“ay ealth facilities within each clust 
ealth centres, dispensaries, etc., and cont; 


B. IN EACH HEALTH FACILITY (HF): 


1. Evaluate the reliability of records ( 


ati : 
cards, etc.). patient registers, 


stock replenishment costs, b: 


2. Note the drugs selec 

ted for the level of he 

: F alth care con 
es at the Beane of inspection? the quantities of each? eae coeepaee. sctumt ie 
replenishment? the date of the last replenishment? : © dates the next 


3. Calculate, based on the n 
umber of treatment epi 
long the stock of each essential drug is going to lace ey The eeaths;: a 
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4. 


Is there in stock drugs which are not selected for this level of health? 


Essential Drugs in stock in each facility 


Type of drug Amount received | Amount distributed Present 
during 3 months during 3 months stock 


— 


How long will 
this stock last 


This table gives answers to two questions: 
= Are the essential drugs selected for the level of health concerned available? 
= Are the stocks going to last till the next distribution? 

C. INTERVIEW WITH PRESCRIBING STAFF/STAFF IN CHARGE OF HF 


Are all essential drugs available now? 
Is the drug supply improving, the same or getting worse? 
How is the level of drug supply in general? 
Are drugs received on a regular basis? 
How frequently? 
Which drugs are always supplied in short? 
-Which drugs are always supplied in abundance? 
Are there any expired drugs in stock? 
For how many weeks (or months) during the past year, have you been out of stock 
of critical drugs (chloroquine, penicilline, ORS vis. )? 
10. Do you think that new drugs must be added to the list? 


e e . 


Omnaur WD eH 
. 


In order to answer questions 6, 7, 8 and 9, the following table should be used: 


Actual stock Out of stock Out of stock Out of stock Comments 


since 1-2 since 3-5 more than including no 


mths (wks) mths (wks) 5 mths (wks) information 
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ample; 
ee coach type of HF of the s p ’ 
A table can be drawn, from the information coll 


this table will facilitate the assessment of dru 


g availability in general: 


No info. 


No. of HFs 
out of stock 
more than 

5 mths (wks) 


No. of HFs 
out of stock 
since 3-5 
mths (wks) 


No. of HFs 
out of stock 
since 1-2 
mths (wks) 


No. of HFs 
with stock 


D. EVALUATE SHORTAGE 


If there is a shortage of essential drugs at the HF level, try to evaluate whether it 
due to: , 

- logistics problems (supply, distribution) (see 4.2) 

os quantification of drugs 

- irrational prescribing or inaccurate diagnosis (see 5.3). 

Evaluate the weight of each component. 


5.2 ACCESSIBILITY OF DRUGS 


The APED coverage will be assessed by measuring the proportion of the population with 
hour's travel to health facilities which are part of the APED programme. 


A. PHYSICAL ACCESSIBILITY 


Physical accessibility indicates, the proportion of the population in the area with 
access to drugs at the health facilities. 


Method 1: Map population distribution in relation to locations at which drugs are 
available. In the absence of population distribution maps, the estimate can be made by 
identifying the location of health facilities providing essential drugs and making a crud 
estimation of the number of people within the defined area, 


Method 2: Evaluate physical accessibility through a one day survey on a sample of 
patients attending health facilities. : 
From this survey information can be obtained on: 
- origin of the patient 
- travel time (in different seasons) 
= number of kilometers 
= mode of transportation. 


Method 33 Examine the records of patients at the health fac 
the geographical distribution of the patients. 
differential utilization rate of patients coming 


ility level and compare w 
(This approach can give a byteroducr: th 
from different distances), 
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B. UTILIZATION 


1. Depending on the available data, try to evaluate: 


the yearly number of people who used ili 
 aaateteb peop used to come to the health facility before the APED 


a the yearly number of patients visiting a heal ili 
th:-£ : : 
Dyed by SEMGREDED \prostamie g acility not being served/being 
= the yearly number of patients at the referral level b 
efor 
programme and now e the start of the APED 


= the yearly number of patients at the referral level in an area not being 
served/being served by the APED programme. 


All these figures will show the impact of the APED programme on the utilization of the 
health facilities and the shift between the treatment at the referral level and primar 
health care level. : 


2. Evaluation of the proportion of the population in need of services (or drugs) who have 
access to the health facilities and who actually do not use them. 


From HF registers, record the number of individuals who contact the health facility (= 
numerator population for assessing coverage). 


It is more difficult to evaluate the population in need of services or drugs but who do 
not use them; the best way is through community surveys (impossible to undertake during an 
evaluation). 


During an evaluation, the only way to assess the utilization rate is to compare the rate 
of utilization of the health facility concerned with a normative utilization rate (average 
number of illness requiring medical care per year) if available. 


3. In order to determine why people for whom the services (including essential drugs) are 
physically accessible, do not use them although they have need for them, try to evaluate the 
socio-economic accessibility of the services: 


Economic barriers 


- Do the people have to pay for the services? 

- Do the people have to pay for the drugs? 

- Are there other ways to obtain drugs freely in the district, mission hospitals, NGOs, 
etc.? 

- Before the APED programme, did they pay for the services? 

- What are the lowest levels of income in the population? 

- Do they have to wait a long time at the health facility? 

- What are the hours of opening of the health facility? 

- Are these hours convenient for the people? 


Socio-cultural barriers 

- Are there many different ethnic groups in the area covered by the health facility? 

- Are there problems between these ethnic groups? 

- Are most of those attending the health facility fromt 

- Is the personnel of the health facility from a particu 
relationship between the number of attendants from one e 
of the personnel at the health facility? : 

~ Is there a common language or many vernacular languages: 

- Does the personnel of the health facility concerned understa 
by the patients? 

- Is there a relations 
and the language spoken by th : 

-~ Does the health facility use translators: 


he same ethnic group? 
lar ethnic group and is there a 
thnic group and the ethnic group 


nd all the languages spoken 


hip between the number of patients speaking a particular language 


e personnel? 
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5.3 PROPER USE OF DRUGS 


For example 


ealth facility? can 
Is feminine 


i the h 
ople using o young? 


certain beliefs which prevent pe 


Are there : nels 
depending on the society concerned: Is the medical eee society where women have 0 
personnel an obstacle to the use of the health facility (\1n 


important social role)? etc. 


Where do they go first when they are sick: 


traditional healers? 
government facilities? 
others? 


How effectively are the essential drugs used in the health facilities serving under AP 


programme? 


A. 


PRESCRIBING HABITS 


Looking at the outpatient registers, what are the prescribing habits of health workers 


like? 

Are drugs dispensed for condi 
antibiotics for a common cold)? 
Are two or more drugs used when one drug would be enough? 

Are drugs generally used for the correct indications? In the correct dosage? 
What training do health workers receive in pharmacology and therapeutics? 

Is this training reinforced in practice? 
Is there a manual for rural health workers to guide them in their clinical diagnosis « 
treatment of patients? If so, is this manual available to every prescriber? 

Are there standard norms for treatment? If so, are they followed? 

Try to compare the treatments given by the health personnel (through out patient 
registers) with the standard treatments included in their manual. 

Try to assess if the training and the manual have led to better diagnosing and 


prescribing practices. 


tions on which they have no therapeutic effects (such as 


DISPENSING METHODS 


What conditions exist at dispensing points? 

How are drugs handled? 

How accurately and cleanly are drugs dispensed? 

Is there a space organized efficiently in order to dispense drugs? 

Is simple equipment available (measuring vials, counting trays, etc.)? 


Who is responsible for the dispensing of drugs? 
What training do these individuals have? 
How much supervision do these people receive? 


 aegain = — = ren to dispense drugs to patients? e.g., individual packet 
n the spot from bulk contai i 
ee iners or course of therapy packets filled out and 
fae a = containers are used for hand dispensing drugs? Ziplock plastic bags 
unsealed plastic bags, paper bags, pieces of paper, bottles, no contai zs 
What type of labeling is used? a 


What type of written instructi i 

ons are given to patients: 
tablets, etc. to be taken), interval, mode of ee. hie 
Is symbolic labeling used? iki 


PATIENT COMPLIANCE 
What is done to increase patient compliance? 


Is there time and space for the dispenser to 


of the drug prescribed? speak with the patient and explain the \ 
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Do health workers in the heal ili isi 
th facilit ; 
can affect patient compliance? y visited know about factors in the culture that 


oi they know the degree to which patients take drugs as prescribed? 
o they believe that patient compliance has improved with the programme? 


What type of community education programmes are promoted? Are group health educati 
sessions conducted? Are health posters related to the use of drugs displ d ion 
walls? In which language? 6 played on the 


- What are the resources available for community education programmes? 
5.4 CLIENT SATISFACTION 
What does the population think about the APED programme? 


= Quantitative assessment 
(See section 5.2.B.) 


= Qualitative assessment has to be done through a questionnaire addressed to patients in a 
sample of health facilities serving under the APED Programme. 


Questionnaire (example) 


- How long have you been coming to this health institution? 
- Have the services improved during the last years? 
- Are drugs more easily available? 
- Do you see evidence that efforts are being made to improve the services? 
- Do you get the kind of drugs you want? 
(are drugs proposed relevant to the perceived priority needs of the popu lation? ) 
- Do you get all the instructions you wish for the use of the drugs prescribed? 
- Can you read the instructions on the labels? 


= In the past, when there were no drugs available at the health facility, from where did 
you obtain drugs? 

- Nowadays, do you still buy drugs outside? 

- Which kind of drugs? 


- When you are sick, where do you go firsts 
- the health facility concerned? 
- the nearest hospital? 
- the traditional healer? 


- others? 
= In the past, before the implementation of the programme, where did you go first? 


- Do you feel that the health activities carried out in your village reflect the needs and 


the priorities of the population? (as expressed by them) 


g the availability of essential drugs 


- Do you feel that these health activities (includin 
he population of the village? 


and vaccines) have improved the health status in t 


5.5 COMMUNITY INVOLVEMENT 


e central authorities or by 
This attitude should be seen as a 


from the programme. 


Very often in programmes planned and controlled by th 
foreigners, the reaction of the communities 18 passive. 
signal that the community does not expect any real progress 
- Describe the mechanisms that exist for community involvement in health at 
regional/district level. 


has rns 8 
=- How have these mechanisms been mobilized for health activities: 


utilization of services and drugs? 


- How do they influence planning (give examples), 
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some questions can be raised: 


In the health facilities visited during the evaluation, 


rkers regularly 
Is there a development committee in the area? If yes, do health wo gZ 
participate? 
does it meet regularly? Does ith @isc 


If yes, 5 
Raise and collect contributions 


- Is there a community health committee? 
formation? 


health activities? Disseminate health in 
health activities (drugs)? 
Does the community help in identifying individuals and groups with specific health 


problems? 


- Does the community participate in the identification of health problems, e.g., lack o 


essential drugs, etc.? 


A more in-depth study will need to be developed for interviews of community leaders. 


5.6 MEDICAL ASSESSMENT OF EFFECTIVENESS 


cult to use for a programme that has not been running for 


This indicator is very diffi ; BS.» 
ts in health are not only due to the availability and 


long time. Secondly, improvemen 
accessibility of essential drugs. 


A. VITAL STATISTICS BY REGIONS 


Crude birth rate Crude death rate Infant mortality rate 
yearn 19 year 19770) year 19. year 19_ | year 19 year 19 


Regions or 


ene: by “eg or district can give some information on the differences 
ent years (with or without APED programm i i 1 
thie Gel. prog e) and between different districts 
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B. INCIDENCE OF SELECTED CONDITIONS REPORTED FOR THE LAST FIVE YEARS 


It can be do i 1 i 
ne mainly at the district/regional level or at the national level 


Diph- Pertus-| Neo- 
Year theria sis natal 
tetanus 


- (i aaa 


All 
tetanus 


Polio | Measles | TBC | Diarrhoea] Malaria 


*criteria will have to be set in order to determine if there is a significant trend in any 
direction for the diseases listed. For example, EPI has used the following criteria in 
programme reviews: 

Significant decrease: reports available for each year, each subsequent year shows less cases 
than the previous year, in the last year the number of cases is less than half the number of 
cases in the first year. 

Moderate decrease: reports available for at least 3 years, tne majority of the reports show 
less cases than the first year, the last year at least 25% less than the first year. 

No decrease: any other situation. 
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6. MANAGING THE ESSENTIAL DRUG PROGRAMME 


6.1 PROGRAMME ORGANIZATION AND MANAGEMENT 

411 be made between the structure over which the programme has direct 

ure through which the programme will have to operate. The former 
inancial management, and the 


1 organization of the programme and f 
rug supply system and the health care delivery system. 


A distinction w 
control and the struct 
relates to the interna 
latter relates to the d 


6.1.1 INTERNAL ORGANIZATION 

committees and management in relation to conte 
(Analysis of Efficiency). 
The 


gress) of 
been dealt with in section 4 
efore merely deal with the management process. 
tion relate to: 


Review of the performance (pro 


(components) of the programme has 
This part of the procedures will ther 
questions to be asked in evaluating the internal organiza 


a): Committees at central level 


What function do the programme committees have? 


Are they meeting regularly? 
- Are long-term issues addressed, e.8-, drug policy and strategy questions? 


- Are minutes issued and followed up? 
- Is there coordination with other programmes, 


e.g.-, EPI, CDD, MCH, etc.? 


b) Committees at regional and district level 


- Have regional and/or district level committees been set up to overlook the programme? 
- What are their functions and how do they perform? 

Do they meet regularly and are field visits included? 

Who do they report to and what is the follow-up procedure? 

- Is there coordination with other programmes, e.g., EPI, CDD, MCH, etc.? 


c) Daily management 


- Is the organizational structure suitable to the circumstances? 
- What are the lines of authority and res ibili i i 
‘ ponsibility and are they in accorda 
established job descriptions? j j ee 
- Is there an informal decision structure? 
- Do the expatriate staff play an advis 
ory role or do they have i 
the problems involved in this? . enh Sabie 
- How is the motivation of the staff (nationals and expatriates)? 
- What are the career prospects and incentives? 
- Does the salary scale cause a problem? 
- Are the national staff likel 
y to seek emplo i i 

a ee ployment in the private sector and what are the 
= Wh . . 
i oon Gis ce a aa reporting system has been established? 

erform thei i ini i 
skills? eir duties or do they need training in management 


- What kind of on-the-job training programme has been established? 


6.1.2 FINANCIAL MANAGEMENT 


The primary tasks of fi ial d t l £ d d t l 
inancia management ar u 5 P i & & 
: : e b = annin unding an contro oO 
operations, and accounting for all resources emp loyed : 


In each of these broad areas, th : 
Poohisticatentn : ’ xe 5 substantial room for applyi i 
aU wilieoee Peivesent’ py So depending on the way the APED erent ee “ies 
Rk as a. goremenent ae on the scale of operations. A pro Oe Genie: | 
Ministries Geammlikely to empl on fiscal principles and rules applicable = FE me 
Bere een flow) ouerces budgeting and einanceal ec. 
analys 


whereas chances of th 
ese tools of financi i 
Ree cet aiPeGeny. c1al management being used increase with the 
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There is little information available in the form of 
activities on these aspects of financial management, 
some basic questions while indicating the direction of 
in this field as results are obtained and APED programm 


actual descriptions of ongoing 

It is therefore proposed to start with 
a somewhat more demanding evaluation 
es enlarge their scope. 


A. BUDGET PLANNING 


This is a catchword for management's role of projecting in a plan of operations the 
required financial resources in the form of recurrent and investment or aa fund 
Gueofulvsllowance:deeds to be made for the consequences, In terms of secu ene Ge, 
arising from past, present and projected investments in construction equipment cies 
and supplies of a useful life exceeding one year. Critical for the aEOceaaaie 
implementation according to plans and commitments is accurate forecasting of the foreign 
exchange component for imports of drugs, associated supplies, fuel, vehicles and parts, and 
for investment in structures where local supplies are lacking. Lastly, budget “oe 
needs to allocate resources by major programme components and in line with the expanding 
regional and local level network. 


The pertinent questions, though subject to local circumstances, are likely to be: 


a) Timely planning and allocations of expected expenditures in the budgets of ministries 
and other public authorities (e.g. national tender boards) active or involved in the APED 
programme; 


b) Due allowance for expenditures to be made in foreign exchange at projected intervals and 
soliciting the actual commitment of external funds under negotiation or likely to be 
forthcoming; 


c) Presentation of budget plan in sufficient detail to distinguish investment from current 
outlays and among major cost categories for accurate accounting and financial performance 
appraisals; 


d) Active search for additional sources of finance, both internally (e.g. development 
banks, national insurance institutions) and externally. Is management active in exploring 
and proposing at the policy level the feasibility and mechanics of cost recovery or 


cost-sharing formulae? 
B. FINANCIAL MANAGEMENT OF DAY-TO-DAY OPERATIONS 


The financial management of day-to-day operations includes the diverse tasks of ensuring 
the availability of funds and releasing payments for expenses such as: purchasing drugs, 
materials and supplies including fuel as well as services, paying salaries, import duties 
and taxes if any, servicing short-term and longer-term debt on due dates and mak ing 
allowance for depreciation of fixed assets by allocating a portion of the investment cost 
representing use, wear and tear during the reporting period to collective or several special 


reserve accounts. 


On the revenue side, the tasks relate in corresponding fashion i) to the billing for 


deliveries and collection of payments due where drugs are being sold, ii) to the handling of 


credits obtained for current operations and capital acquisitions, and iii) to claims against 
suppliers arising from non-observance of contract terms. 


efficient management of the scheme as a whole are tested by 
ent operations are being handled. Procurement 
ernationally and astute placing of orders in a rhythm 
ed inventory requirements are probably the most _ 
eminent cost-containing measures on which to focus. More demanding are fits RE 
with detailed projections and subsequent monitoring of capital ee ene s; wi pate 
t including short-term investments of liquid balances and hedging against tore1g 
Trees riske: and with continuous financial performance control by cost centre and 
eions such. ad procurement, treasury, billing and collections and supervision of loca 


outlets. 


Financial discipline and 
evaluating first and foremost how curr 
procedures to obtain lowest prices int 
jn line with likely turnover and relat 
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ations can be derived 
ion to test the groun 
jent but also m 


erformance in current oper 
time the occas 


developing more effic 


g the record on p 
ions and provide at the same 


to management's interest in 


Questions establishin 
from the above considerat 
during the evaluation as 
demanding methods and tools. 


Cc. ACCOUNTING AND AUDITING 
ection deals with accounting in a double sense: operating a reliable and 
ting for all funds entrusted to management in t 


counts, and accoun 
to supervisory and auditing bodies as well as to external sponsor 
nt questions are likely to be: 


This last s 
transparent system of ac 
form of regular reports 
of the programme. The pertine 
loped to track transactions, inventories an 


a) Is the accounting system sufficiently deve 
ts and down to local outlets? 


al equipment in use by programme componen 


capit 
rative cost analyses by products, local outlet 


b) Does the accounting system enable compa 
and user groups? 


c) Which improvements are now being made or are feasible to ensure the above? 
ting and distribution rules and procedures are in use or could 
s of cost containment within standard frames and of safeguarc 


dary or black market? 


d) What purchasing, accoun 
improved to meet requirement 
against leakage of drugs into a secon 


e) Which institutional modifications appear required to strengthen the financial managen 
role at central, regional and local levels? 


£) For which functions and at which levels is training in financial management skills me 
urgently needed? What training facilities, programmes and instructional materials are it 


use and could be built upon? 
g) Is the programme being audited? by whom? how often? 
6.1.3 DRUG SUPPLY SYSTEM 


a . . ~ 
” ... s hale structure and relationship between the different units i 
upply system (procurement unit), tender b d i i 
a > oard, therapeutic committee lannin 
inance units, import and inventory control units, medical stores at national os ag 
regional/district and health care level. 


b : ; ; 
) Describe the functions and responsibilities of the different units 
c) Assess the ; 

performance of each unit separately and in relation to the total structur 


ad) 5G 
ggest ways to remove bottle-necks and bureaucratic procedures 


6.1.4 HEALTH CARE DELIVERY SYSTEM 


It is not the pur : 
pose of this evaluati 
care personnel but onl : ion to assess the general perform 
y their performance in relation to drug use Be kines = 
’ cribdil 


and dispensing practices) Vv 
- However, thi i i 
ns P bats me eS s has been dealt with in other parts of these 
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6.2 BUDGETING, FINANCING AND COSTING 


This section will revi en ure 
ew the structure and trend 
4 s of health exp 
sae = ways of improving allocation patterns and generating crea — 
analysis may point the way to possible cost savings and ways of ntreaqianie 
& 


» with a view to 
rees. Such Jan 
Operations. 


6.2.1 PATTERNS OF HEALTH EXPENDITURE 


a) Present a table of expenditures, both planned and actual, by instituti 

of Health) for each year of the period under evaluation, py aiib6 nosaieee oe see Bi 
preceding the beginning of the programme. This should permit any i items , 
noted. Distinguish between local and government resources and external alee —_ 
to determine how much of the external assistance is on a grant basis, and ho sai ace 
to be repaid (if possible, note the interest rates of each source). (ne thes plat 
into local currency (using IMF exchange rate information from International Sinanoialia 
Statistics). Note significant differences between planned and actus leaeGcee ae 
Significant increases or decreases in any given category of expenditure over time (fee 
example, a significant drop in MOH budget allocations), and attempt to explain these (dro 

in price of main export, oil price rise, etc.). é 


Ministry 


b) Analyse capital and recurrent expenditures of MOH budget by categories (personnel, 
drugs, materials, fuel and transport, maintenance, etc., including pro rata shares of 
capital depreciation for buildings and other significant investments, if possible). 


c) Analyse the role and importance of the private sector, where applicable. Include 
estimates of expenditures on drugs, private medical care including hospitalization, etc. 
Include estimates of expenditures of Social Security institutions if applicable. Although 
difficult to obtain, rough estimates of the importance of military medical and health care 
in terms of numbers of beneficiaries, number of staff employed, drug purchases, etc. may be 
very useful (may include police as well in many countries). 


d) With regard to external assistance, what types of projects and assistance have been 
provided? Have recurrent costs been adequately estimated and planned for? Is it possible 
to discern trends as to likely continuation and levels of assistance? Attempt to determine 
how much of the external assistance is on a grant basis, and how much will have to be 
repaid. Convert external assistance amounts into local currency (use IMF exchange rate 
information from Intl. Financial Statistics). (UNDP is usually an up-to-date and convenient 
source of information on external assistance.) Identify any external assistance in the area 
of drug supply and provide as mich specific information as possible as to amounts of 


funding, activities, etc. 


6.2.2 MANAGEMENT AND OPERATIONAL ASPECTS 


a) Assess the reliability and use by management of financial accounting, control, and 
11 be on the Ministry of Health and the 


auditing information. Focus in most countries wi 
agency responsible for drug importation and distribution (Central Medical Stores 6 
equivalent). What practical steps can be taken to improve performance in this area? 


b) Describe the situation with regard to foreign exchange. How scarce 1s create ; 
exchange? What is the mechanism for release of foreign exchange, and on what ae a 
foreign exchange released? What opportunities exist for achieving api in pe 
exchange? Are there any special arrangements with regard to oe aE et a fl grins i 
with specific countries, such as the CFA franc zone and the French franc: y 


these. 


i i i thorizing 
z r am, tracing the major steps 1n budgeting, au 
See ae, otc. % rented of the time required at each step. Pay 


, ; ; t1 
payments, making disbursements, etc. with es What could be done to shorten 


i i hase of drugs. 
tention to the procedures for purc 
sage 4 ‘red for each step? 


the pipeline and reduce the time require 


Who decides on this? What margin is applied? How do these 


in the private sector? 


are dru rices set? 
. at f similar products 1 


prices compare with prices 0 
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6.2.3 ASSESS ING CosTS 
transparency among components and 


h for the sake of 
for continuous improvements OF fo 


This is the most critical task-—-bo : 
and as a basis 


functional areas of the current programme ? d in a manner 
costing alternative strategies. Costs need to be disaggregated and allocate 


i i i) by type o 
reflecting actual occurrence in operations. Most pertinent are cost analyses oe fe yp 
administrative levels, iii) by output in te 


; ii tivit t various ee bee 
inputs, ii) by activity a lons and iv) finally by user group Or individual user. 


services rendered at various eche 
ns, it is now easier to 
d indirect cost items, calculat 


d thus establish signposts 


Based on the earlier analysis of structure and operatio 
ell as direct an 


distinguish between fixed and variable as w 
various average and incremental (or so-called marginal) costs an 


for a smoother flow of funds in operation. 


sis is employed to show how objectives of soc 
roached and met. 


Comparative cost analy ial equity, equal 
access, geographical equilibrium are being app 
and actual application of funds by various levels of 


a) Tabulate budget appropriations 
lth centre dispensing) and by functional cost 


service delivery (e.g. hospital versus hea 
centres (i.e. procurement, storage, distribution). 
us actually incurred costs for drugs by 
source (procurement) through 

final dispensing and compare with earlier 
say, more than 20 per cent of total 


b) Present and interpret data for budgeted vers 
therapeutic class; trace the cost build-up from 
process/handling stages with typical mark-ups to 
assumptions by concentrating on drugs accounting for, 
turnover. 


c) What kind and portions of indirect costs borne by the health care system at large can 
allocated to the essential drug programme? Explain approach and rationale. 


d) Tabulate comparative costs by drug category, service levels, regions and in terms of 
urban/rural users to highlight equity aspects of the programme. 


= “lice are the costs incurred by households to obtain access to service (i.e. transport 
ealth centre, time lost for work) and what could be done to reduce these further? 


£ ; ‘ 
aa a do previously estimated and actual annual per capita costs for essential drugs an¢ 
pharmaceuticals at large compare? Current projections and rationale? 


g) What is the approximate outla i 
y for drugs in absolute terms and as i 
care outlays by households in various regions and by income/wealth sl eas ili a 


K) oH é : 
) How much does this present in terms of disposable income of households? 


j) What are the respective meri 
erits a i ; 
heey F197 Nae nd problems with revenue generation by sale of drugs & 


k) What proportion of their i 
or their income do household i n n 
other prescribed medicines, and on freely availabe eta eree Le tliat. 
uticals. 


Several recent WHO documents i 
provide explanato i 
souaere. ot ; ry detail and po 3 . 
Pp Gr Seecssing costs, €.g., reference (5) (15) annex 7 hee eee oe 
5) an 
page 80 
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6.3 MONITORING 


Monitoring consists of th i i 
e timely collection and revi 
ext ; ew of a numb 
ancvcators NAGS when taken together, provide an adequate index of Ne order ee apes. 
a Fi 1on through the execution and operation phases. In other words, the aéatheee 
any BaPOxt whether inputs, work schedules (activities), outputs and other One aes ode 
2 a Rs pai ibang to plan. The monitoring system is mainly designed to provide he 
ement of the programme with the information needed to f iviti 
ollow- 
a up activities on a 


When evaluating a monitoring system it is first of all important to assess wheth h 
chosen indicators describe the programme, objectives and implementation aspects fo cs : 
targets, expectations and basic assumptions regarding each objective and aeneoet Ror ; 
be monitored. Secondly, the processing and analysis of data should be eva lane a x 
thirdly, it is necessary to assess whether the monitoring results are presented 5 a 
digestible way and used in the decision-making process. 


6.3.1 INDICATORS 
The chosen indicators should have the following characteristics: 


a) They should cover the different aspects of implementation and operation of the 
programme. The indicators should provide information on the efficiency (chapter 4) and 
effectiveness (chapter 5) of the programme. 


b) They should be simple, and the number of indicators should be limited. The idea is not 
to have indicators which describes the total universe of the programme, but rather a few 
indicators which produce the information necessary for decision-making. 


c) They should be cost-effective. Just as it is not always necessary to have a separate 
indicator for each aspect of the programme, it is also not necessary to have full coverage 
of any particular factor. For example, a sample at a few health centers can be sufficient 
instead of collecting data continuously at every health center. 


d) They should be adapted to the manpower and financial resources of the country and, in 
particular, of the programme. Monitoring should not be based on complex and expensive data 
collection techniques that would be difficult to apply in the country ona continuous basis. 


6.3.2 PROCESSING AND ANALYSIS OF DATA 


In other monitoring systems the following major pitfalls have been experienced in the 
processing and analysis of data. Check to see if they also apply to the APED programme. 


a) lack of detailed planning 

b) lack of funds 

c) poor distribution of resources between data collection on one hand, 
analysis on the other hand 

d) establishment of unrealistic deadlines for producing monitoring reports 

e) too much data, which is usually a consequence of poorly selected indicators and/or 


overambitious sample size 

£) poor questionnaire design or inappropria 
to the data processing capacity 

g) lack of delayed development of software an 
time-consuming validation processes 


h) not enough computer time 
i) inadequately trained personnel, 


and processing and 


te tailoring of the format of the questionnaire 


d analysis procedures, as well as 


insufficiently skilled staff for doing manual 
tabulations, lack of highly skilled expertise for computer processing of the data, 


overload of assignments, and high turn-over of staff 
3) field-level checking of data not institutionalized. 
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6.3.3 PRESENTATION AND USE OF MONITORING RESULTS 


results is as important as the design of indicators an: 


nd analysis of data. If reports are poorly written and/or presented, and ver} 


have the impact they deserve. The presentation should therefore 
at is issue- and problem-oriented. The 


The presentation of monitoring 


processing 4 


elaborate, they will not 
avoid the above mistakes and include a summary th 
whole process of maintaining a monitoring system is useless if the results are not used by 


the management in decision-making. Confront management with some of the problems brought 
in the monitoring reports and check to see if the information has been used and what acti 
they have yet to do. This test is very important because it says something about the 


usefulness of the monitoring system and points to correction in the design. 
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7. ASSESSMENT OF INPUTS OF THE GOVERNMENT AND DONORS 


= sections a. 5 and 6 the evaluation h i ed l b £ 
é . , as consider a arge number o 
ection wl 


deal with the in 
puts or rather the management/administrative 
e g / istrati support from the Government, 


The following questionnaire can be used to indicate whether performance duri 

period covered by the evaluation has been negative, as anticipated or su nee ep a 

the factor has played an important role in programme performance. Ae ns whether 
not applicable should be so marked. Pertinent factors which do not appear in ae iia 


questionnaire should be added in the spaces provided, as sh ae 
explanatory comments. P P ’ should any clarifying or 


Performance vs. 
expectations 


v vu 
SJ Co) 
-O 7a) 
Ls) 9) 
1] Au 
“d 4h 
cal d) 
Q “4 
Qu, a) 
© (= 
oO 
pS) 
° n 
a < 


cv) 
> 
S| 
=) 
oO 
oO 
vo 
a 


7.1 GOVERNMENT 


a. Support in preparing programme documents 


b. Guidance on programme background 


c. Guidance on national political/institutional situatio 
d. Guidance on policy and legal aspects 
e. Guidance on administrative formalities 


f. Support in undertaking administrative formalities 


g. Support in recruitment of local staff 


h. Support in selection of local consultants 


i. Support in obtaining supplies 


j- Payment of salaries/allowances/expenses 


k. Provision of physical facilities and equipment 


1. Availability of secretarial help 
m. Provision of office material/equipment 
n. Provision of transport facilities 


o. Provision of storage facilities 


Maintenance and repair of equipment 


Other factors and/or comments: 
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7.2 WORLD HEALTH ORGANIZATION 


Performance vS. 


ions 


expectat 


As anticipated 


Superior 


Not applicable 


Negative 


Support in preparing technical parts of programme 
documents 

Support in developing technical strategy 

Overall adequacy of technical backstopping 

Support in preparing programme documents/revisions 


Guidance/assistance on design/evaluation methodolog | 


Timely provision of programme funds 

Response to requests for assistance 

Payment of salaries/allowances/ expenses 

Coordination with related programmes (EPI,CDD, PHC) 

Timely recruitment of international staff 

Timely recruitment of international consultants a 

Timely procurement of drugs, equipment and dressings arn 

Timely placement of fellows | 
= 


Timely reply to administrative consultations 


Guidance on policies and procedures of WHO | a 


Other factors and/or comments: 
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Performance vs. 


expectations 


if important 


7.3 DONOR(S) 


us) 
v 
v 
iS) 
Qu 
‘4 
S) 
od 
‘s) 
= 
is) 
n 
<= 


Not applicable 
Negative 
Superior 
Check, 


a. Support in developing technical strategy Sieve 2 n:s0scalgg a ae 
b. Support in preparing programme documents/revisions Le |. | 
c. Clarity/appropriateness of procedural guidelines 1 —E—— ——ee 
d, Timely provision of programme funds pes’ |: yl Siena obs ae 
e. Response to requests for assistance ee EE ee eee 
f. Support in recruitment of international staff per oc | Ten oe: sa 
g. Support in recruitment of international consultants Ear” 
h. Timely placement of fellows Peecie 34 


i. Timely procurement of drugs, equipment and dressings 


j- Timely replies to administrative consultations he | ee 


k. Guidance on policies and procedures of donor 


1. Coordination with related programmes (EPI ,CDD , PHC) Wa ine pee 


Other factors and/or comments: 


7.4 OTHER PARTICIPATING AGENCIES 


a. Timely procurement of drugs, equipment and dressings . | i 
b. Timely replies to administrative queries >. |_| a 
c. Usefulness of replies to queries Le ae 


Other factors and/or comments. 
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8, PLANNING FOR THE FUTURE 


inui j nsidered. 
In the event the prograinme is continuing, the following areas should be co 


8.1 PROGRAMME PLANN ING 
the plan of operation and work 


Based on the overall conclusions and recommendations, 


schedules should be revised accordingly. 


8.2 PROGRAMME OBJECTIVES 
ginal programme objectives and immediate 


When revising the plan of operation the ori 
the future needs. 


objectives (components) should correspond with 


8.3 PROGRAMME STRATEGY 


e.g., is there a need to shift the 


Will the strategy remain the same in the future, 
pleted and therefore must be 


balance between the components or has one component been com 
deleted, etc.? 


8.4 MANAGEMENT 


Based on the revised Programme Plans, Objectives and Strategy is there a need to chan; 
the organizational set-up of the programme? Must terms of reference for committees and }j 


descriptions for individuals be rewritten? 


8.5 FEASIBILITY 


If the evaluation recommends an expansion or extension of the programme, then it is v 
important to analyse whether the programme will be economically and financially feasible 
the Government in the future. In earlier sections of these procedures, the efficiency of 
the programme has been analysed within the present programme set-up. In planning for the 
future, sources of finance in the form of, e.g., cost recovery and price policies might h 
to be considered in order to make the programme affordable for the Government. By "cost 
recovery" is meant a system whereby a portion of the cost of drugs and/or a consultation 
borne by the patient. By "price policy" is meant a system whereby cost recovery is based 
the patient's ability to pay and/or the relative importance of the drugs. 


8.6 EXTERNAL ASSISTANCE 


Finally, the need for future assistance from WHO and/or other donors should be 
assessed. Future assistance might take a different form than when the programme was 
designed and probably should, because the country in the long run will have to cover all 
public expenditures for the programme itself. 


17. 


18. 
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considerable preparation is mandatory. 
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PHASES AND STEPS OF AN APED EVALUATION 


In general, an APED evaluation (hereafter called evaluation), is carried out in an 
intensive team effort over a relatively brief time period (in the order of three weeks). 
However, to prepare for the optimal use of the evaluation team during that period, 


from the evaluation are proportional to the quality and the thoroughness of 


are 


and 


In fact, one might argue that the benefits derived 


the planning. 


For that reason, the evaluation is partitioned into six phases, the first three of which 


devoted to the planning. These phases are; 


A. Initiation Phase 

B. Design Phase 

Ca Administrative Preparation Phase 

D. Field Work ; 

E. Analysis Phase, including planning for the future 
F. Documentation Phase. 


In turn, each phase is partitioned into steps; some pertaining to concept and others 
pertaining to logistics. The discussion of each phase will include a listing of the steps 


a commentary of variable length on each step. The reader is cautioned t 


o approach the 


application of the steps with an open mind - each needs to be applied judiciously within the 
context of the given evaluation. The steps and the implied sequence are pre 


through the programme evaluation process, mot as 4 prescription which must b 


costs. 
A. INITIATION PHASE 


The steps in the jmitiation phase are; 


he Take the decision to proceed with the evaluation based on assessmen 
the country and the usefulness of an evaluation in meeting those ne 


2. Assign responsibility for the conduct of the evaluation. 


mo ‘ : di 
3. Identify supporting and participating agencies, both national and 1 


4. Set the dates for the evaluation. 


sented as a guide 
e followed at all 


t of the needs of 
eds. 


nternational. 
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aluation. Prior to initiating an evaluation, 


ision to proceed with the ev a 
7 a 4 —— the aoeiis of the country be identified and that careful consideration 
Be oe i 1 eeting those needs. In the past, evaluatioi 


e role an evaluation might play in m 


i th f i ati 
eae se they provide an opportunity: 


have been quite useful becau 
entation of an essential drugs 


implem 
easure the progress made towards imp | ; 
extent to which well defined goals for improvement 


accessibility of drugs, proper use of drugs, clien 
lvement have been achieved. 


(a) tom 
programme in terms of the 
of availability of drugs, 
satisfaction and community invo 


(b) to identify obstacles to continued progress, and identify those elements whi 


are developing well and according to plan; 
(c) to identify areas where research needs to be conducted in order to improve t 
programme ; 


(d) to validate routinely monitoring data regarding programme implementation in 


the country; and 


(e) to gather data not normally a part of a country's information system, 
especially data on the process of implementation not easily codified in 


routine data collection systems. 


2. Assign responsibility for the evaluation 


In order to increase the probability of success of an evaluation, it is advisable to 
assign clear responsibilities for the evaluation to appropriate individuals and the steeri 
committee of the programme. 


3. Identify supporting and participating agencies 


One factor which has contributed to the success of completed evaluations is the 
diversity and breadth of the team ultimately charged with carrying out the evaluation. 
Early on, it is advisable to identify all international agencies and any non-governmental 
organizations who play a role in the delivery of essential drugs. These groups should be 
informed of the intention to do the evaluation and, where appropriate, efforts should be 
made to secure commitments as to their participation. 


4. Set the dates for the evaluation 


It is imperative that the dates of the evaluation are set as early as possible. Becai 
of the many scheduling difficulties and logistical obstacles which arise in any programme 
evaluation, it is essential that dates be fixed early in the process so that work on 


resolving the practical problems associated with the evaluation can proceed in a definitiv 
way. 


= A most important date is that of the post-evaluation meeting (section E.3) at which 
igh-level officials act on the findings of the evaluation in reformulating policy and 


os he babe the plan of action for implementation. The commitment of the appropriate 
eee Sal a pana time to the post-evaluation meeting must be secured well 
e meeting is to be held on schedule and without inte i 
rfer 
demands of government service. ie: - ames 


B. DESIGN PHASE 

The steps in the design phase are: 
Review and make a preliminary analysis of existing information. 
Dis Collect information about the programme. 


Jie Collect data relating to the achievement of stated ob 


performance. jectives and project 


4, Draw the sample for assessment of effectiveness 
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5. Design the data collection (survey) instruments. 


6. Field test the survey instruments, 


1. Review and make a preliminary analysis of existing information 


It is important to assemble a great deal of information 
an evaluation. A first step is the gathering of general inf 
health care system (see section 1 in "PROCEDURES'"') 


prior to the field work phase of 
ormation of the country and the 


A second step should be collection of macroe i i 
conomic and i i 
programme (see section 3 in "PROCEDURES"). ilatitlil io cap sie 


2. Collect information about the programme 


All the information about the programme should be 
: collected bef i 
field work phase. This might include: efore the beginhing oF “the 


a) Policy statements, circulars and ordinances re i isti 
: garding the programme 
for implementation, etc. (see section 2 in PROCEDURES"). wen eesen ane 


b) Programme objective and immediate objectives, inputs, activities, outputs, 
indicators and assumptions of each component of the programme (see section 4 in 
"PROCEDURES" ). 


c) Any past evaluations of the programme or its separate components 


d) All relevant monitoring data. 


3. Collect data relating to the achievement of objectives and project performance 


Considerable time will be saved by the evaluation team if data related both to the 
achievement of objectives (as reflected by selected indicators) and project performance (as 
reflected by inputs achieved, activities undertaken, and outputs produced) are collected 
beforehand. Objectives, assumptions, indicators, etc. should not be altered at this stage, 
either in order to reflect the current status or in anticipation of the evaluation findings. 


4. Draw the sample for assessment of effectiveness 


The selection of a sampling scheme requires that a balance be struck between cost and 
precision and that the scheme is feasible given logistical and time constraints. The 
proposed assessment of effectiveness (see section 5 in "PROCEDURES") is not very detailed 
and can be carried out in a short period (7 days) at the health centre/dispensary level. 
Should it be decided at a later stage to conduct an evaluation at all levels of the health 
care system, e.g. community level (village health workers, community leaders, home level), 
health centers/dispensaries and district hospitals, the question of drawing a sample becomes 


much more complicated. 


heme is a prerequisite for determining the 
During that phase, enough staff must be on hand 
Estimates of staff required can be 
eir relative accessibility are 
ds can be fixed only after the 


Note, the selection of the sampling sc 
manpower needed for the field work phase. 
to complete the field work within the designated time. 
made only when the number of sites to be visited and th 
known. Also, transportation requirements and budgetary nee 


sampling scheme is defined and the staff needs determined. 


5. Design the survey instruments 


Generally, this step "pins down 


* * te 
‘eld data to be gathered. Within the con 
peg to balance i breadth of each instrument against the depth, the desire for 


i i i f forded 


structured coded questions. 


" the exact nature of the evaluation by de fining exactly 
xt of a programme evaluation, it is 
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6. Field test the survey instruments 
developed untri 
Such a test should: 


If a survey is to use newly ed instruments of data collection, then the: 
should be field tested first. 


rvey instruments, especially the 


the package of su 5 
e the entire process oO 


a) confirm the feasibility of 
d the timing required to complet 


clarity of each question an 
data gathering; 


entire package of survey instruments in a 


b) confirm the feasibility of using the rvey 
r logistical constraints. 


selected geographical area given the timing and othe 


Clearly, the field test could lead to appropriate modification of the original survey 


instruments. 


C. ADMINISTRATIVE PREPARATION PHASE 


The first two phases deal more with the development of the conceptual framework for th 
evaluation. In this third phase, the more mechanical aspects of the evaluation are attend 
to. As we noted earlier, these mechanical aspects are as critical to the success of the 

evaluation as are the conceptual aspects. Therefore, we urge that this phase begin as ear 


as possible; in fact, much of this phase can be done in parallel with the first two. The 
steps in the administrative preparation phase are: 


1) Form the team which will conduct the evaluation. 

2) Forma proposal as to how to partition the larger team into smaller survey teams. 
3) Make all the logistical arrangements for the field work. 

4) Secure the required national and international financial support. 

5) Arrange for training and working facilities at central and provincial levels. 


6) Arrange for translators, guides and, most importantly, secretarial support for tl 
entire field visit phase. 


1. Form the team which will conduct the evaluation 


; tae _ first pertinent administrative decisions is the formation of the evaluatio 
eam. is decision is taken considering issues relating to b i 

_Th oth the size 
composition of the team. : = 


As the evaluation is technical, the iti mb 
os ’ composition should reflect this. 
a minimum, have the following background: Members should, 


--physician 
--health educator 
--health economist 
--sociologist 
--pharmacologist 


and be drawn from the following groups: 


~-Ministry of Health 
~“Ministry responsible for procurement of drugs 
Piers responsible for local production 


atte : : ; ; 
rnational agencies playing a role in the development of the programme 


> 1 Sg 


the team cannot i 

consist solely of internati 

: i 

country evaluation. onal or external agency personnel as it is a 


DAP/85.8 
MANUAL page 5 


Finall note th i i ici i 
of tee Seger aie ee Participation must be arranged and confirmed far ahead 
>£ the aluation. Such arran i i fi 
ees gements should be with specif 
uals within the international agencies as well as with the agencies NERVES 


2. Partition the evaluation team into smaller survey teams 

The larger evaluation team should be 
team must act in parallel with the others 
the gathering of data in the field. 


partitioned into smaller Survey teams. Each survey 
» but as a self-contained independent unit during 


3. Make all logistical arrangements 


Once again, we reiterate the absolute requirement to arrange all logistical aspects of 
the field work in advance. In this step, we emphasize the more mundane arrangements 
including: 


a) hotel bookings, both central and in the field; 
b) vehicle reservations, including petrol and drivers, and; 
c) within country airline reservations, if needed. 
4. Secure the required national and international financial support 


Programme evaluation of the programme is not an inexpensive activity. If the evaluation 
is not part of the regular budget of the programme, other sources must be sought, e.g., 
national and/or international resources. 


In estimating costs, one might consider using the form in Annex l. 


A comprehensive budget should be prepared, and revenues should be identified as to 
source. 


5. Arrange for working facilities at central and provincial levels 


In addition to arranging all of the logistical aspects of the survey (step 3) p°Re Cie 
necessary to arrange the facilities for the survey teams to do their "office" work (data 
tabulation, analysis and report writing). The teams will need facilities at both central 


and provincial levels. 


6. Arrange for translators, guides and, most importantly, secretarial support for the 


entire field visit 


members with documents prepared 


Translators may be needed to assist international team : 
eeded to assist in the field work 


in local languages. More importantly, translators may be n 
in areas where local dialects serve as primary languages. 


all phases of the field visit - not merely for report 
begin preparing draft materials at the start of 
for later use in the final report. 


Secretarial support is critical at 
writing at the end. Often, evaluation teams 
the evaluation for both intra-team exchanges and 


D. FIELD WORK PHASE 


is envisioned as a short but intensive phase carried out by the 


The field work phase 
It consists of only two steps: 


entire evaluation team. 


tof d 
1) Conduct surveys at the health center/dispensary level and visit the regionan au 


district health authorities. 


2) Perform a "first level" data analysis. 
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i isi ional and 
£ Conduct surveys at the health center/dispensar level and visit the re 
istrict health authorities 
district 1° 


i i i a at ine 
A major activity in the evaluation process 1S the esc eaete of a — 

i ional and district hea 
r/dispensary level, but the region alt 292 2aBs* 

Se ae to get a full picture of the programme's efficiency and effect 


2, Perform a "first level" data analysis 


The "first level" data analysis consists of the summarization (in most cases, the 
totalling) of the responses to each question from the field survey. 


E. ANALYSIS PHASE 
The analysis can be achieved in four steps. 
1) Conduct the second level data analysis. 


2) Prepare a preliminary report, including data summaries, findings, conclusions and 
recommendations. 


3) Prepare the plan of action for implementing short-term solutions. 


4) Present and discuss preliminary conclusions and recommendations with 
decision-makers (post-evaluation meeting). 


1. Conduct the second level data analysis 


Upon returning from the field, the team should create master files of the survey 
instruments and the summaries generated during the first level data analysis. The second 
level data analysis consists of the compilation of the relevant summaries of the entire da 
collection activity from the preliminary summaries or, where necessary, from a return to t 
original survey instruments. 


2. Prepare a preliminary report, including data summaries, findings, conclusions and 


recommendations 


A preliminary report should be prepared as quickly as possible including summaries of 
the relevant data, interpretations, findings and conclusions derived from the summaries, 
and recommendations. Data should be presented in simple tables, using easily understood 
modes of presentation. Consolidation of the data into fewer variables is desirable. The 
source for all data should be cited and all tables should be titled and clearly labelled. 


3. Prepare the plan of action for implementing short-term solutions 


One of the most valued outcomes of the evaluation is a_plan of action for implementin 
short-term solutions. Preparation of this plan should be done by the evaluation team its 


4. Present and discuss preliminary findings, conclusions, recommendations and plan of 
action with decision-makers (Post-evaluation meeting) 

The success of the programme evaluation is reflected in the response of decision-make 
to the pertinent findings. The magnitude and quality of that response are likely to be 
enhanced if the evaluation team reviews all its findings, conclusions and recommendations 
with the appropriate decision-makers before the team is disbanded. 


F. DOCUMENTATION PHASE 


The documentation phase consists of one step: 


document. This can be done more slowly, 
field visit. 


the preparation of the formal evaluat; 

after the evaluation team has ended its i 

an b Ss intense 

ate Le 1s important to prepare a formal report to create a record of the 

pave to provide a tool for those charged with implementing the recommendations of 
10n, and to leave a basis for subsequent evaluations. Moreover, in the event tha 


| 


II. TIMING 
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ANNEX 1 


1.2 


5 


3. 


SUGGESTED BUDGET FRAMEWORK 


Preparato phase 
Evaluation preparatory meeting: 
Meeting facilities 
Participants 
Travel cost 
Secretarial support 
Stationary/miscellaneous supplies 
OtherS cecccvccccccccsssesers 
Field testing of methodology/survey forms: 
Participants 
Travel cost 
Stationary/printing forms 
Duplication/distribution of background doc. 
Other expenses 
Evaluation 


Evaluation team meeting: 


Meeting facilities 

Participants 

Travel cost 
Hospitality/miscellaneous 
Field survey: 

Daily allowance 

Internal travel 

Rental of vehicles 

Petrol,oil, lubricants,maintenance,repairs 
Drivers, guides, interpreters 
Others 

Stationary and administrative cost 
Compilation/distribution of draft report 
Secretarial support 

Publication of final report 
Compilation 

Translation 

Editing 

Printing 

Distribution 

Others 


TOTAL 


Additional contribution from bilateral/ 


multilateral agencies 


Grand total 
Number of consultants person/weeks funded by 


———— 


Proposed 
Budget sources 
of 
funds 


10. 


ll. 


12. 


13. 


14. 


TIMETABLE OF ACTIVITIES DURING THE EVALUATION 


Activity 
Assembly of evaluation team members 
Meeting of evaluation team 


Assignment of team members to field 
visits 


Explanation of objectives, methods 
and instruments 


Administrative preparations for field visits 


Collection of data at the central level 


Tabulation of central level data and preparation 
of short report on findings at Central level 


Field visits 


- travel to field 

> training of interviewers 
= collection of data 

- tabulation of data 

- travel back from field 


Plenary session and presentation of field 
team reports (Identification of achievements 
and problems and tabulation of data) 


Dividing into subgroups to discuss relevant areas 
Preparation of lists of achievements, problems 
and recommendations with a plan of action with 


timetable for implementation for each area 


Preparation of draft summary of findings, 
conclusions and recommendations 


Plenary session to summarize findings, 
conclusions and recommendations 


Presentation of summary of findings, 
conclusions and recommendations 


to Ministry of Health 


Preparation of draft final report 
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ANNEX 2 


5. to 13 


id 


PentnwUN 
(a 
° 


14 (half day) 


15 to 18 


19 


20 


21 


22 onward 


mestione) con ribucion from 
Matlieceral egencins 

¥ total 

mer Of conan! ents 


b 


J a 


oe 


T ALE 


te 


ato 31 
tla 


> 
ay) 
dam 


ts 


